FILM: Following Instructions in Language with Modular Methods
Authors: So Yeon Min
Devendra Singh Chaplot
Pradeep Kumar Ravikumar
Yonatan Bisk
Ruslan Salakhutdinov
Venue: ICLR 2022 Poster (2022)
Keywords: Instruction Following
Visual Language Navigation
Embodied Instruction Following
VLN
ALFRED

Abstract: Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

Published as a conference paper at ICLR 2022
FILM: FOLLOWING INSTRUCTIONS IN LANGUAGE WITH MODULAR METHODS
So Yeon Min
Devendra Singh Chaplot
Pradeep Ravikumar'
Yonatan Bisk'
Ruslan Salakhutdinov
CCamegie Mellon University
Facebook Al Research dchaplot@ @hhaplot@fb.com
{soyeonm, pradeepr, ybisk, rsalakhu] @cs.cmu.edu
ABSTRACT
Recent methods for embodied instruction following are typically trained end-to- end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semanties to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose modular method with structured representations that (1) builds semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46%) with a substantial (8.17 % absolute) gap from previous work while using less data by es- chewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49%). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions."
INTRODUCTION
Human intelligence simultaneously processes data of multiple modalities, including but limited to natural language and egocentric vision, in an embodied environment. Powered by the success of machine learning models in individual modalities (Devlin et al., 2018; He et al., 2016; Voulodimos et al.; Anderson et al., 2018a), there has been growing interest to build multimodal embodied agents that perform complex tasks. An incipient pursuit of such interest was to solve the task of Vision Language Navigation (VLN), for which the agent is required to navigate the goal area given language instruction (Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020).
Embodied instruction following (EIF) presents more complex and human-like setting than VLN or Object Goal Navigation (Gupta et al., 2017: Chaplot et al., 2020b; Du et al., 2021); beyond just navigation, agents required execute sequences of sub-tasks that entail both navigation and interaction actions from a language instruction (Fig. 1). The additional challenges posed by EIF are threefold the agent has understand compositional instructions of multiple types and subtasks, choose actions from a large action space and execute them for longer horizons, and localize objects fine-grained manner for interaction (Nguyen et al., 2021).
Most existing methods (Zhang & Chai, 2021; Kim et al., 2021; Nottingham et al., 2021) for EIF have relied on neural memory of various types (transformer embeddings, LSTM state), trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs. However, EIF remains very challenging task for end-to-end methods as they require the neural net simultaneously learn state-tracking, building spatial memory, exploration. long-term planning, and low-level control.
In this work, we propose FILM & (Following Instructions in Language with Modular methods). FILM consists of several modular components that each (1) processes language instructions into
'The official ALFRED leaderboard: http:////ctttoooaaaaaiii//aaaaas
Projecct wehpage with code and pre-trained models: https://socoomm.iifhh../o//// webpage/

---

Published as a conference paper at ICLR 2022
(8)
Instruction:
Put heated apple on a counter.
Semantic Search
Policy
Apple
Sink
Semantic Mapping Module
Micro
wave
Countertop
(b)
Subtask: Pick Up Apple
Subtask: Microwave Apple
Subtask: Put Apple on Countertop
An Embodied Instruction Following (EIF) task consists multiple subtasks. (a) FILM method overview: The agent receives the language instruction the egocentric vision of the frame. every time step, a semantic top-down map of the scene is updated from predicted depth and instance segmentation. Until the subgoal object is observed, search goal (blue dot) is sampled from the semantic search policy. (b) Example trajectories: Trajectory of existing model (HiTUT (Zhang Chai, 2021)) plotted in straight green line, and that of FILM is in dotted red. While HITUT's agent travels repeatedly over path of closed loop (thick green line, arrow pointing in the direction of travel), FILM's semantic search allows better exploration and the agent sufficiently explores the environment and completes all subtasks.
Figure 1:
structured forms (Language Processing), (2) converts egocentric visual input into semantic metric map (Semantic Mapping), (3) predicts a search goal location (Semantic Search Policy), and (4) outputs subsequent navigation/ interaction actions (Deterministic Policy). FILM overcomes some of the shortcomings of previous methods by leveraging a modular design with structured spatial components. Unlike many of the existing methods for EIF, FILM does not require any input that provides sequential guidance, namely expert trajectories or low-level language instructions. While Blukis al. (2021) recently introduced method that uses structured spatial memory, comes with some limitations from the lack of explicit semantic search and the reliance on expert trajectories.
On the ALFRED (Shridhar et al., 2020) benchmark, FILM achieves State-of-the-AA performance (24.46%) with a large margin (8% absolute) from the previous SOTA (Blukis et al., 2021). Most approaches rely on low-level instructions, and we too find that including them leads an additional 2% improvement in success rate (26.49%). FILM's s strong performance and our analysis indicate that an explicit structured spatial memory coupled with a semantic search policy can provide better trackin and exploration, even in the absence of expert trajectories or low-level instructions.
RELATED WORK
plethora of works have been published on embodied vision and language tasks, such as VLN (Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020), Embodied Question Answering (Das al., 2018: Gordon al., 2018), and topics of multimodal representation learning (Wang al., 2020: Bisk et al., 2020), such as Embodied Language Grounding (Prabhudesai et al., 2020). For Visual Language Navigation, which is the most comparable the setting of our work, methods with impressive performances (Ke et al., 2019; Wang et al., 2019; Ma et al., 2019) have been proposed since the introduction of R2R (Anderson et al., 201 8b). While far from conquering VIN,, these methods have shown up 61% success rate unseen test environments (Ke et al., 2019).
For the more challenging task of Embodied Instruction Following (EIF), multiple methods have been proposed with differing levels of modularity in the model structure. As a baseline, Shridhar et al. (2020) has presented Seq2Seq model with an attention mechanism and a progress monitor, while Pashevich et al. (2021) proposed to replace seq2seq model with episodic transformer. These methods take the concatenation of language features, visual features, and past trajectories as input and predict the subsequent action end-to-end. On the other hand, Kim al. (2021); Zhang
2

---

Published as a conference paper at ICLR 2022
& Chai (2021); Nguyen et al. (2021) modularly process raw language and visual imputs into struc- tured forms, while keeping separate "action prediction module" that outputs low-level actions given processed language outputs. Their "action taking module" itself is trained end-to-end and relies on neural memory that "implicitly" tracks all of spatial, progressive, and states of the agent. Unlike these methods, FILM's structured language/ spatial representations make reasons for failure transparent and elucidates directions to improve individual components.
Recently, Blukis et al. (2021) has proposed more modular method with persistent and structured spatial memory. Language and visual input are transformed into respectively high-level actions and the 3DD map. With the 3D map and high-level actions as input, low-level actions are predicted with a value-iteration network (VIN). Navigation goals for the VIN are sampled from model trained on interaction pose labels from expert trajectories. Among all proposed methods for EIF, FILM necessitates the least information (neither low-level instructions nor expert trajectories are needed, although the former can be taken as additional input). Furthermore, FILM addresses the problem of search/ exploration of goal objects.
Various works in visual navigation with semantic mapping are also relevant. Simultancous Local- ization and Mapping (SLAM) methods, which build 2D or 3D obstacle maps, have been widely used (Fuentes-Pacheed et al., 2015; Izadi et al., 2011; Snavely et al., 2008). In contrast these works, recent methods (Chaplot et al., 2020b;a) build semantic maps with differentiable projection operations, which restrain egocentric prediction errors amplifying in the map. The task of Chaplot et al. (2020b;a) is object goal navigation, a much simpler task compared to EIF. Furthermore, while Chaplot al. (2020b) employs a semantic exploration policy, our and their semantic policies serve fundamentally different purposes; while their policy guides a general sense of direction among mul- tiple rooms in the search for large objects (e.g. fridge), ours guides the search for potential locations of small and flat objects which have little chance of detection at a distance. Also, our semantic policy conditioned on language instructions. Blukis et al. (a;b)) also successfully utilized semantic 2D maps in grounded language navigation tasks. These works are for quadcopters, whose fields of view almost entirely cover the scene and the need for "search" or "exploration" " is less crucial than for pedestrian agents. Moreover, their settings only involve navigation with single subtask.
3 TASK EXPLANATION
We utilize the ALFRED benchmark. The agent has to complete household tasks given only natural language instructions and egocentric vision (Fig. 1). For example, the instruction may be given as "Put a heated apple on the counter," with low-level instructions (which FILM does not use by default) further explaining step-by-step lower level actions. In this case, one way to "succeed" in this episode is to sequentially (1) pick up the apple, (2) put the apple in the microwave, (3) toggle the microwave on/off, (4) pick up the apple again, and (4) place it on the countertop. Episodes run for significantly longer number of steps compared to benchmarks with only single subgoals; even expert trajectories, which are maximally efficient and perform only the strictly necessary actions (without any steps to search for an object), are often longer than 70 steps.
There are seven types of tasks (Appendix A.1), from relatively simple types (e.g. Pick & Place) to more complex ones (e.g. Heat & Place). Furthermore, the instruction may require that an object is "sliced" (e.g. Slice bread, cook it in the microwave, put it on the counter). An episode is deemed "success" if the agent completes all sub-tasks within 10 failed low-level actions and 1000 max steps.
4 METHODS
FILM consists of three learned modules: (1) Language Processing (LP), (2) Semantic Mapping, and (3) Semantic Search Policy; and one purely deterministic navigation/ interaction policy module (Fig. 2). the start of episode, the L.P module processes the language instruction into sequence of subtasks. Every time step, the semantic mapping module receives the egacentric RGB frame and updates the semantic map. If the goal object of the current subtask not yet observed, the semantic search policy predicts a "search goal" at a coarse time scale; until the next search goal is predicted, the agent navigates to the current search goal with the deterministic policy. If the goal is observed, the deterministic policy decides low-level controls for interaction actions (e.g. "Pick Up" object).
4.1 LANGUAGE PROCESSING (LP)
The language processing (LP) module transforms high-level instructions into a structured sequence of subtasks (Fig. 3). It consists of two BERT (Devlin al., 2018) submodules that receive the in-
3

---

Published as a conference paper at ICLR 2022
Instruction
Subtasks
Semantic Search Policy
Language Processing
Drop a clean pan the table
(Pan, PickUp). (SinkBasin, Put). (Faucst. ToggleOn). (Faucet, TffgleOff). (Pan, PickUp), (Table, Put)
Search Goal
Egocentric RGB
Semantic Map
Semantic
Mapping
Deterministic
Pollcy
Action
RotateRight
Figure 2: FILM method overview. The "grouping" in blue, green, and yellow denote the coarseness of time scale (bluc: at the beginning of the episode, green: at every time step, yellow; at a coarser time scale of every 25 steps). At the beginning of the episode, the Language Processing module processes the instruction into subtasks. every time step, Semantic Mapping converts egocentric into RGB a top-down semantic map. The semantic search policy outputs the search goal at coarse time scale. Finally, the Deterministic Policy decides the next action. Modules in bright green are learned; the deterministic policy (grey) is not.
struction as input the heginning of the episode. The first submodule (BERT type classification) receives the instruction and predicts the "type" of the instruction one of the seven types stated in Appendix A. The second submodule (BERT argument classification) receives both the instruc- tion and the predicted type as input and predicts the "arguments"" (1) "obj" for the object to be picked up. (2) "recep" for the receptacle where "obj" should be ultimately placed, (3) "sliced" for whether "oby "obj" should be sliced, and (4) "parent" for tasks with intermediate movable receptacles (e.g. "cup" in "Put a knife in cup on the table" of Appendix A. An object in ALFRED is always an instance of either "obj" or "recep"; "parent" objects are a subset of "recep" objects that are movable. We train separate BERT model for each argument predictor. The two submodules are casily trainable with supervised learning since the type and the four arguments are provided in the training set. Models use only the CLS token for classification, and they do not share parameters; all layers of
were fine-tuned.
Due to the patterned nature of instructions, we can match the predicted "type" of the instruction to "type template" with blank arguments. For example, if the instruction is classified as the "clean & place" type. it is matched to the template "(Obj. PickUp), (SinkBasin, Put). (Faucet, ToggleOn), (Faucet, ToggleOff). (Obj. PickUp), (Recep, Put)". If the "sliced" argument is predicted to be true from argument classification, subtasks of "(Knife, PickUp), (Obj, Slice), (Sink, PutOhject)" will be added at the beginning of the template (with the (Sink, PutObject) make the agent drop the knife). Filling in the "type template" with predictions of the second model, we obtain a list of subtasks (bottom of Fig. 3b) be completed in the current episode. The "type templates" were designed by hand in less than 20 minutes. In section 5.2, we discuss the effect of using LP module without the template assumption, for fair comparison with other works. Appendix A.9 contains more details,
4.2 SEMANTIC MAPPING MODULE
We designed the semantic mapping module (Appendix A.2) with inspirations from prior work (Chaplot et al., 2020b). Egocentric RGB is first processed into depth map and instance segmen- tation, with MaskRCNN (He al., 2017) (and its implementation by Shridhar et al. (2021)) and the depth prediction method Blukis et al. (2021): ;; details the training are explained in Section 5 finetuned on the training scenes of ALFRED. Once
These pre-trained, off-the-shelf models processed, the depth observation is transformed to a point cloud, of which each point is associated with the predicted semantic categories. Finally, the point cloud binned into a voxel representation; this summed over height is the semantic map. The map is locally updated and aggregated over time.
The resulting semantic map is an allocentric (C + 2) M M binary grid, where C is the number of object categories and each of the M x M cells represents 5cm x 5cm space of the scene. The C channels each represent whether particular object of interest was ob- served; the two extra channels denote whether obstacle exists and whether exploration happened We use the publicly released code of Shridhar al. (2021); Blukis al. (2021).
4

---

Published as a conference paper at ICLR 2022
BERT
(b)
Instruction
Drop cloan pan table
type classification
Subtask template
(1 for each of the types) (Ohj, PicKUp). (SinkBasin, Put). (Faucet, ToggloOn), (Faucet, ToggleOr), (Obj, PickUp), (Recep, Put)
BERT
argument classification
Instruction Type| of types
Place
Subtasks
Instruction Arguments
Object: Pan, Recep: Table,
(Pan, PlckUp), (SinkBasin, Put), (Faucet, ToggleOn), (Faucet, ToggleDrr,, (Pan, PiekUp), (Table, Put)
Parent: None,
Sliced: False
Figure 3: The Language Processing module. (a): Two BERT models respectively predict the "type" and the "arguments" of the instruction. (b): The predicted "type" from (a) is matched with a template, and the "arguments" of the template is filled with the predicted "argument."
in a particular 5cm x Sem space. Thus, the C + channels are a semantic/spatial summary of the corresponding space. We use M 240 (12 meters in the physical world) and C 28 + (number of additional subgoal objects in the current task). "28" the number of "receptacle" objects (e.g- "Table", "Bathtub"), which are usually large and easily detected; in the example of Fig. I, there is one additional subgoal object ("Applc"). Please see Appendix A.2 on details of the dynamic handling of C.
4.3 SEMANTIC SEARCH POLICY
The semantic search policy outputs a coarse 2D distribution for potential locations of small sub- goal object (Fig. 4), given a semantic map with the 28 receptacle objects only (e.g. "Countertop", "Shelf"). The discovery
small object is difficult ALFRED due to three reasons - (1) many objects are tiny (some instances of "pencil" occupies less than 200 pixels even very close view), (2) the field of view is small due to the camera horizon mostly being downward", (3) semantic seg- mentation, despite being fine-tuned, cannot detect small objects at certain angles. The role of the semantic search policy is to predict search locations for small objects, upon the observed spatial con- figuration of larger ones. While existing works surmise the "implicit" learning of search locations from expert trajectories, we directly learn an explicit policy without expert trajectories.
The policy is trained via supervised learning. For data collection, we deploy the agent without the policy in the training set and gather the (1) semantic map with only receptacle objects and (2) the ground truth location of the subgoal object after every 25 steps. model of 15 layers of CNN with max-pooling in between (details in Appendix A.3) outputs an N x N grid, where N is smaller than the original map size M; this a 2D distribution for the potential location of the subgoal object. Finally, the KI. divergence between this and a aseudo-ground truth "coarse" distribution whose mass is uniformly distributed over all cells with the true location of the subgoal object is minimized (min, K (pll4) where p is the coarse ground truth and & is the coarse prediction). At deployment, the "search goal" is sampled from the predicted distribution, resized to match the original map size of M M (e.g. 240 240), with mass in the coarse N x N (e.g. 8 x 8) grid uniformly spread N x L area centered on it. Because arriving at the search goal requires time, the
out to the | M policy operates at a "coarse" time scale of 25 steps; the agent navigates towards the current search goal until the next goal is sampled or the subgoal object is found (more details in Section 4.4).
Fig. 4 shows visualization of the semantic search policy's outputs. The policy provides reason- ably close estimate of the true distribution; the predicted mass of "bowl" is shared around observed furniture that can appear on, and that of "faucet" peaks around the sink/ the end of the bathtub. While we chose N = 8 as the grid size, Appendix A.4 provides a general bound for choosing N.
4.4 DETERMINISTIC POLICY
Given (1) the predicted subtasks, (2) the most recent semantic map, and (3) the search goal sampled at a coarse time scale, the deterministic policy outputs navigation or interaction action (Fig. 2).
Let ((bbj, action)), ... (obje, actiong)] be the list of subtasks and the current subtask be (obje, action.). If obj; is observed in the current semantic map, the closest obj; is selected as the goal: otherwise, the sample from the semantic search policy is chosen as the goal (Section 4.3). The agent then navigates towards the goal via the Fast Marching Method (Sethian, 1996) and performs
"The agent mostly looks down 45" in FILM for correct depth prediction. Looking down is common in existing models well (Kim et al., 2021; Zhang & Chai, 2021; Blukis al., 2021).
5

---

Published as a conference paper at ICLR 2022
Loofeee
Side
Rowl
Tollet
Shelf
Safe
Fauoet
Shefff
Figure 4: Example visualization of semantic search policy outputs. In each of (a), (b), Top left: map built from ground truth depth/ segmentation, Top right: map from learned depth/ segmentation, Bottom left: ground truth "coarse" distribution, Bottom right: predicted "coarse" distribution. (a): While the true location of the "bowl" was on the upper Ieft coffee table, the policy distributes mass over all furniture likely to have it on. (b): The true location of the faucet is on the sink and at the end of the bathtub. While the policy puts more mass near the sink, also allocates some to the end of the bathtub.
the required interaction actions. While this "low-level" policy could be learned with imitation reinforcement learning, we used a deterministic one based on the findings of earlier work that ob- served that the Fast Marching Method performs as well as learned local navigation policy (Chaplot et al., 2020b). When the agent successfully executes the required interaction action, (which can be determined by the change the egocentric RGB), the pointer of subtasks is advanced to + or the task is completed. More details and pseudocode are provided in Appendix A.5.
EXPERIMENTS AND RESULTS
We explain the metrics, evaluation splits, and basclines against which FILM is compared. Further- more, we describe training details of each of the learned components of FILMM
Metrics Success Rate (SR) is a binary indicator of whether all subtasks were completed. The goal- condition success (GC) a model the ratio of goal-conditions completed at the end of an episode. For example, in the example of Fig. 1, there pan should rest on a countertop. and a "clean" pan should rest on a countertop. Both SR and GC be weighted by (path length of the expert trajectory)/ (path length taken by the agent); these called path length weighted SR (PLWSR) and path length weighted GC (PLWGC).
three coal-conditoons
pan must be "cleaned",
Evaluation Splits The test set consists of "Tests Seen" (1533 epsiodes) and "Tests unseen" (1529 episodes); the scenes of the latter entirely consist of rooms that do not appear in the training set, while those of the former only consist of scenes seen during training. Similarly, the validation set is partitioned into "Valid Seen" (820 epsiodes) and "Valid Unseen" (821 epsiodes). The official leaderboard ranks all entries by the SR on Tests Unseen.
Baselines There are two kinds of baselines: those that use low-level sequential instructions (Kim al., 2021; Zhang & Chai, 2021; Nguyen al., 2021; Pashevich al., 2021) and those that do (Nottingham et al., 2021; Blukis et al., 2021). While FILM does not necessitate low-level instruc- tions, we report results with and without them and compare them against methods of both kinds.
Training Details of Learned Components In the LP module, BERT type classification and argu- ment classification were trained with Adam W from the Transformer (Wolf et al., 2019) package; learning rates are 1e-6 for type classification and (Ie-4.1--5.5e-55ee55] for each of "object", "par- ent", "recep", "sliced" argument classification. In the Semantic Mapping module, separate depth models for camera horizons of 45 and 0 were fine-tuned from an existing model of HL.SM (Blukis et al., 2021), both with learning rate 1e-3 and the Adam optimizer (epsilon 1e-6, weight decay le-2). Similarly, separate instance segmentation models for small and large objects were fine-tuned, starting from their respective parameters released by Shridhar al. (2021), with learning rate le-3 and the SGD optimizer (momentum 0.9, weight decay 5e-4). Finally, the semantic search policy was trained with learning rate 5e-4 and the AdamW optimizer (epsilon le-6). Appendix A.2 and A.3 discuss more details on the architectures of semantic mapping/ semantic search policy modules. The readme of our code states protocols and commands that readers can reproduce all expriments.

---

Published as a conference paper at ICLR 2022
Table I: Test results. Top section uses step-hy-step instructions; bottom section does not. Bold numbers are top scores in each section. Blue numbers are the top SR on Tests Unseen (by which the leaderboard is ranked). Method
Tests Seen
Tests Unseen
PLWGC
GC
PLWSR
SR
PLWGC
GC
PLWSR
SR
Low-tevel Sequential Instructions + High-level Goal Instruction
SEQ2SEQ
(Shridhar et al., 2020)
6.27
9.42
2,02
3.98
4.26
7.03
0.08
MOCA
(Singh ct al., 2020)
22.05
28.29
15.10
22.05
9.99
14.28
2.72
5.30
E.T.
(Pashevich et al., 2021)
36.47
28.77
15.01
5.04
E.T. + synth. data
(Pashevich et al.. 2021)
34,93
45.44
27.78
38.42
11.46
18.56
4.10
8.57
LWIT
(Nguyen et al., 2021)
23.10
40.53
43.10
30.92
16.34
20.91
5.60
9.42
HITUT
(Zhang & Chai, 2021)
17.41
29.97
11.10
21.27
11.51
20.31
5.86
13.87
ABP
(Kim et al., 2021)
4.92
51.13
3.88
44.55
2.22
24.76
1.08
15.43
FILM SEMANTIC SEARCH
13.10
35.39
9.43
25.90
13.37
35.51
10.17
23.94
FILM E
15.06
38.51
11.23
27.67
14.30
36.37
10.55
26.49
High-level Goal Instruction Only
LAV
(Nottingharn et al., 2021)
13.18
23.21
6.31
13.35
10.47
17.27
3.12
6.38
HITUT G-only
(Zhang & Chai, 2021)
21.11
13.63
17.89
11.12
HLSM
(Blukis et al., 2021)
11.53
35.79
6.69
25.11
8.45
27.24
4.34
16.29
FILM W.O. SEMANTIC SEARCH
12.22
34.41
8.65
24.72
12.69
34.00
9.44
22.56
FILM E
14.17
36.15
10.39
25.77
13.13
34.75
9.67
24.46
5.1 RESULTS
Table shows test results. FILM achieves state-of-the-art performance across both seen and un- seen scenes in the setting where only high-level instructions given. It achieves 8. 17% absolute (50.15% relative) gain in SR on Tests Unseen, and 0.66% absolute (2.63% relative) gain in SR on Tests Seen over HI.MM, the previous SOTA.
FILM performs competitively even compared to methods that require low-level step-by-step in- structions. They can be used additional inputs to the LP module, with the lowelevel instruction appended to the high-level instruction for both BERT type classification and BERT argument classi- fication. Under this setting, FILM achieves 1 11066% absolute (71.68% relative) gain in SR on Tests Unseen compared to ABP. Notably, FILM performs similarly across Tests Seen and Tests Unseen, which implies FILM's strong generalizability. This is in contrast to that methods that require low- level instructions, such as ABP, E.T., LWIT, MOCA, perform very well on Tests Seen but much less so on unseen scenes. In a Sim2Real situation, these methods will excel if the agent can be trained in the exact household it will be deployed in, with multiple low-level instructions and expert trajecto- ries. In the more realistic and cost-efficient setting where the agent is trained in a centralized manner and has to generalize to new scenes, FILM will be more adequate.
also notable that the semantic search policy significantly increases not only SR and GC, but also their path-length weighted versions. On Tests Seen, the gap of PLWSR between FILM with/ without semantic search is larger than the corresponding gap of SR (for both with/ without low-level instructions). This suggests that the semantic policy boosts the efficiency of trajectories. The results in Appendix A.8 show that the improvement by the semantic policy is reproduced across multiple seeds; the protocols for reproduction are explained along with the result.
5.2 ABLATIONS STUDIES AND ERROR ANALYSIS
Errors due to perception and language processing. To understand the importance of FILM's individual modules, we consider ablations on the base method, the base method with low-level language, and with ground truth visual/ language inputs. Table shows ablations on the development sets. While the improvement from gt depth is large in unseen scenes (10.64%), it is incremental on seen scenes (1.48%) ); on the other hand, gt segmentation significantly boosts performances in both cases (9.26% / 9.26%)) Thus, among visual perception, segmentation is a bottleneck in both seen/ unseen scenes, and depth is bottleneck only in the latter. On the other hand, while a large gain in SR comes from using ground truth language (7.43 % / 4.22 %), that from adding low-level language as input is rather incremental. We additionally analyze the effect of the template assumption (explained in the second paragraph of Section 4.1), by reporting the performance with Language Processing module without this assumption. The results drop without the templates but not by a large margin. Appendix A.9 explains the details of this auxiliary Language Processing module.
Error modes. Table 3 shows common error modes of FILM; the metric is the percent of episodes that failed (in SR) from particular error out of all failed episodes. The main failures in valid unseen scenes are due to failures in (1) locating the subgoal object (due to the small field of view, imperfect segmentation, ineffective exploration), (2) locating the subgoal object because it is in closed

---

Published as a conference paper at ICLR 2022
Table 2: Ablation results on validation splits. Base Method FILM with semantic search policy.
Table 3: Error Modes. Table showing per- centage of errors due to each failure mode for FILM the Val set.
Val Seen
Val Unseen
Method
GC
SR
GC
SR
Error mode
Seen
Unseen
Base Method
37.20
24.63
32,45
20.10
Goal object not found
23.30
26.07
Interaction failures
8.54
low-level language
38.54
25.24
32.89
20.61
Collisions
6.96
11.00
gl seg.
45.46 46
34.02
42.88
29.35
Object closed receptacle
18.44
16.16
gt depth
38.21
26.59
42.91
30.73
Language processing
18.53
24.54
gi depth, seg.
55.54
43.22
64.31
$5.05
Others
25.81
13.69
gt depth, gt seg., lang.
59.47
47.44
69.13
62.48
template assumption
31.46
20.37
31.14
18.03
Table 4: Performance by task type of base
valldd unsean
model on validation.
valld
Val Seen
Val Unseen
Task Type
GC
GC
SR
Overall
37.20
24.63
32.45
20.10
Examine
50.00
34.41
45.06
29.65
Examine
Stack
Plck
Clean &
Heat & Place
C Place
2 Place
Pick Place
27.46
26.92
16.67
16.03
Light
Place
Stack & Place
23.74
10.71
9.90
1.98
Figure Average number subtasks completed until fail- ure, by task type (light green/ light blue respectively for valid seen/ unseen). Dark green/ blue: average number of total subtasks in valid seen/ unseen.
Clean Place
58.56
44.04
48.89
33.63
Cool & Place
27.04
12.61
27.41
14.04
Heat Place
40.21
22.02
37.77
23.02
Pick 2 & Place
40.37
23.77
29.28
11.84
ceptacle (cabinet, drawer, etc), (3) interaction (due to object being too far or not in field of view, bad segmentation mask), (4) navigation (collisions), (5) correctly processing language instructions, (6)
others, such as the deterministic policy repeating a loop actions from depth/ segmentation failures and 10 failed actions accruing from a mixture of different errors. A failed episode is classified to the error type that occurs "earlier" (e.g. If the subtasks were processed incorrectly and also there were 10 consecutive collisions, this episode is classified as (5) (failure in incorrectly processsing language instructions) since the LP module comes "carlier" than the collisions). More details are in Appendix A.6. As seen in Table 3, goal object not found is the most common eπror mode, typically due to objects being small and not visible from a distance or certain viewpoints. Results of the next subsection show that this error is alleviated by the semantic search policy in certain cases.
Performance over different task types. To understand FILM's strengths/ weaknesses across dif- ferent types of tasks, we further ablate validation results by task type in Table 4. Figure 5 shows the average number subtasks completed for failed episodes, by task type. First, the SR and GC for "Stack & Place" is remarkably low. Second, the number of the subtasks entailed with the task type does not strongly correlate with performance. While "Heat & Place" usually involves three more subtasks than "Pick Place", the metrics for the former are much higher than those of the latter. Since task types inevitably occur in different kinds of scenes (e.g. "Heat & Place" only occurs in kitchens) and therefore involve different kinds of objects (e.g. "Heat & Place" involves food only), the results suggest that the success of the first PickUp action largely depends on the kinds of the scene and size and type of the subgoal objects rather than number of subtasks.
While the above error analysis is specific to FILM, its implications regarding visual perception may generally represent the weaknesses existing methods for EIF, since most recent methods (ABP, HLSM, HiTUT, LWIT, E.T.) use the same family of segmentation/ detection models as FILM, such as MasK-RCNN and Fast-RCNN (Wang et al., 2017). Specifically, it could be that the inability to find a subgoal object is major failure mode in the mentioned existing methods well. On the other hand, FILM is not designed to search inside closed receptacles (e.g. cabinets), although subgoal objects dwell in receptacles quite frequently (Table 3); future work to extend FILM should learn to perform a more active search.
5.3 EFFECTS OF THE SEMANTIC SEARCH POLICY
Table Dev set results (valid unseen) of FILM with/ without semantic search policy.
With Valid Unseen the development set, observed that the semantic search policy significantly helps to find small objects (Table 5) use the percent of episodes in which the first goal object was found (% 1st Goal Found) proxy, since it can be picked up (e.g. "Apple", "Pen"")
Method
1st Goal Found
HLSM (Blukis et al.. 2021)
N/A
11.8
FILM with Search FILM w.o. Search
80.51
20.09
76.12
19.85
8

---

Published as a conference paper at ICLR 2022
Instruction: Put a large clean knife on the counter
Go to! Search for "Knife'
"Knife" "Countertog"
Pick up "Knife"
to "SinkBasin"
Wash "Knife"
Plox up "Knifs"
Egocentric RGB
144
1-114
t=316
1=324
1-330
3333
FILM with Search
FILM W.O. Search
Figure 6: Example trajectories of FILM with and without semantic search policy. Paths near the subgoals that traveled 3 times more in straight red. The goal (which can be the search goal observed instance of subgoal object) is in blue.
Table 7:
Performance with and without semantic scarch policy, by task type.
Table 6:
Performance with and without semantic search policy, by room size.
Clean & Place
Other Types
Room Size
Small
Large
Task Type
FILM
FILM
FILM
FILM
FILM
FILM W.O. Search
FILM
FILM w.o. Seurch
woo. Search
w.o. Search
SR
% 1st Goal Found 79.32
26.70
26.63 81.02
15.17 80.13
14.74 73.72
% 1st Goal Found
87.61
33.63
14.16 79.65
79.38
17.94
75.56
20. 16
% 1st Recep Found 80.53
69.03
58.05
55.93
and thus is usually small. Thus, we use FILM with semantic search as the "base method" (default) for all experiments/ ablations.
To further analyze when the semantic search policy especially helps, we ablate on room sizes and task types. Table 6 shows the SR and 1st Goal Found with and without search, by room size (details on the assignment of Room Size are in Appendix A.7). As expected, the semantic policy increases both metrics, especially so in large scenes. This is desirable since the policy makes the agent less disoriented difficult scenarios (large scenes); the model without it is more susceptible to failing even the first subtask. Figure 6 is consistent with the trend of Table 6; it shows example trajectories of FILM with and without the semantic search policy in a large kitchen scene. Since the countertop appears in the bottom right quadrant of the map, it is desirable that the agent travels there to search for a "knife". While FILM travels to this area frequently (straight red line in Fig.6), FILM without semantic search mostly wanders in irrelevant locations (e.g. the bottom left quadrant).
Table 7 further shows the performance with and without search by task type. Notably, the gap of performance for the "clean & place" type is very large. In the large kitchen scene of "Valid Unseen" (Fig. 6), the "Sink" looks very flat from a distance and is hardly detected. The semantic policy induces the agent to travel near the countertop area and improves the localization of the 1st Recep (""innk") for the "clean & place" type (Table 7). In conclusion, the semantic policy improves the localization of small and flat objects in large scenes.
6 CONCLUSION
We proposed FILM, new modular method for embodied instruction following which (1) processes language instructions into structured forms (Language Processing), (2) converts egocentric vision into a semantic metric map (Semantic Mapping), (3) predicts likely goal location (Semantic Search Policy), and (4) outputs subsequent navigation/ interaction actions (Algorithmic Planning). FILM achieves the state of the art the ALFRED benchmark without any sequential supervision.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This research is for building autonomous agents. While we do not perform any experiments with hu- mans, practitioners may attempt to extend and apply this technology environments with humans. Such potential applications of this research should take privacy concerns into consideration.
All learned models in this research were trained using Ai2Thor (Kolve et al., 2019). Thus, they may be biased towards North American homes.
REPRODUCIBILITY STATEMENT
We thoroughly explain training details and model architectures in Section 5.1 and Appendix A.2, A.3. Project webpage with code, pre-trained models, and protocols reproduce results is released here: https//soreoom..gttub../LLLL. git.hub.bo/LLMM..baagee.
REFERENCES
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi. Manolis Savva, al. On Evaluation of Embodied Navigation Agents. arXiv preprint 1rXiv:/807.067555 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. lision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018b.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds lan- guage. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing (EMNLP), 2020.
Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A Knepper, and Yoav Artzi. Following high- level navigation instructions simulated quadcopter with imitation learning. In Robotics: Science and Systems (RSS), 2018a.
Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position-vin visitation pp. 505-518. PMLR, 2018b.
position-visitation prediction, In Conference Robot Learning,
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. persistent spatial se- mantic representation for high-level natural language instruction execution. In Proceedings of the Conference on Robot Learning (CoRL). 2021.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. arXiv preprint arXiv:2004.055 05155,, 2020a.
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using gral-oriented semantic exploration. Advances Neural Information Processing Systems, 33, 2020b.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruy Batra. Embod- ied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810. 04805, 2018.
Heming Du, Xin Yu, and Liang Zheng. Vtnet: Visual transformer network for object goal navigation.
arXiv preprint arXiv:2 105 004404444444.0000 2021.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kinn Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. follooer models for vision-and aasooo--nd-aangaae -anguuge navigation. In Advances in Neural Information Processing Sys- fems, 2018
10

---

Published as a conference paper at ICLR 2022
Jorge Phentes-PPaceco. José Ruiz-Ascencio, and Juan Manuel Gendón-Manche Visual simultane- ous localization and mapping: a survey. Artificial intelligence review, 43(1):55-88) 011:555-11220 2015.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4089-4095 2018.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings the Vision and Pattern Recognition, pp. 2616-2625, 2017.
the IEFE Conference on Computer
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cun. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium User interface software and technology, pp. 559-568, 2011.
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking vision-and- language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp.
2019.
Byeonghwi Kim, Suvaansh Bhambri, Kunal Pratap Singh, Roozbeh Mottaghi, and Jonghyun Choi. Agent with the big picture: Perceiving surroundings for interactive instruction following. In Embodied Al Workshop CVPR, 2021.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequencceto-sequuecce pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 7871-7880, On- line, July 2020, Association for Computational Linguistics. doi: 10. Nt.18653/v1/2000 ac1220.cllmmn 1/2205511/0222.llmmmm URL http::/wclanttollg..o//200.0222 m03..703.
Chih-Yes Yao Ma. Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 1732-6740 2019.
Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. Look wide and interpret twice: Improving performance interactive instruction-following tasks. arXiv preprint wrXiv:2/06.005%6 2021.
Kolby Nottingham, Litian Liang, Dachyun Shin, Charless C. Fowlkes, Roy Fox, and Sameer Singh. Lav, 2021. URL https//leaderboard hwtp://wwww.rloobbbiiio/////e/ea
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. arXiv preprint arXiv:2 10l.:223550 2021.
Mihir Prabhudesai, Hsiao- Yu Fish Tung, Syed Ashar Javed, Maximilian Sieb, Adam W Harley, and Katerina Fragkiadaki. Embodied language grounding with 3d visual feature representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2220- 2020.
11

---

Published as a conference paper at ICLR 2022
J A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences, 93(4):1591-1595, 1996. ISSN 0027-8424. doi: 10. 10.1033/pnas. 93.4.1591. URL https:/wwww..pmssooo/////mmtt//s
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Mohit Shridhar, Xingdi Yuan, Alex-nle Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Moca: A modular object-centric approach for interactive instruction following. arXiv preprint 2tt::20122.3208.. 2020.
Noah Snavely, Steven M Seitz, and Richard Szeliski. Modeling the world from internet photo collections. International journal of computer vision, 80(2):189-210, 2008.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: brief review. Computational intelligence and neuroscience, 2018.
Ruocheng Wang, Jiayuan Mao, Samuel Gershman, and Jiajun Wu. Language-mediated, object- centric representation learning. arXiv preprint arXiv:2012.158/1 2020.
Xiaolong Wang, Abhinav Shrivastava, and Abhinav Gupta. A-fast-renn: Hard positive generation via adversary for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2606-2615, 2017.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan- Tuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised im- itation learning for vision-language navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6629-6638, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ard natural language processing. arXiv preprint arXiv: 2019.
Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-moniniee monitoring. arXiv preprint arXiv:2106.03447, 2021.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10012-10022, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A. TASK DEFINITION
High and low-level instructions are both available to agents. There are 7 types of tasks (Fig 7. b) and the sequence of subtasks templated according the task type.
Instruction Goal: Drop the table.
Goal: lowards gas, pick up pan.
sink,
Step 0
Step L
Step 1.2
Step L_3
Step
Step t.5
t.6
Step 1_7
"pan" "sink"
Toggle
Toggle "Faucet"
Pick up "Pan" & navigato to "Table"
Put "pan" on "Table"
Navigate
Hick "pan"
*Faucet"
(b)
Pick & Place
Stack & Place
Pick Two & Place
Clean & Place
Heat & Place
Cool
Examine
& Place
in Light
Put a candle on the toilet
a in a cup
Put two remole the couch
Pul two remole on the couch
Place heated apple sink.
potato siiee
grey bowl in the light of lamp
gerbage.
Figure ALFRED overview. The goal is given in high level and low level language instructions. For agent to achieve "success" of the goal, it needs to complete a sequence of interactions (as in the explanations in the hottom of the figure) and the entailed navigation between interactions.
A.2 SEMANTIC MAPPING MODULE
Figure is illustration of the semantic mapping module. A depth map and instance segmentation is predicted from Egocentric RGB. Then the first and the later are respectively transformed into point cloud and semantic label of each point the cloud, together producing voxels. The voxels are summed across height to produce the semantic map. Partial maps obtained at particular time steps are aggregated the global map simply via "sum/ logical or."
Projection Map X.M X.M
R-CNN
Labels
Depth Prodiction
Point Cloud
Votal
Doeatung Natach
Semanic Map Predction (K M.M.
Figure 8: Semantic mapping module. Figure partially taken from Chaplot al. (2020b)
We dynamically control the number of objects C for efficiency (because there are more than 100 objects in total). All receptacle objects (for input to the semantic policy) and all non-receptacle objects that appear in the subtasks are counted C. For example, in an episode with the subtask [(Pa,, PickUp), (SinkBasin, Put), (Faucet, ToggleOn), (Faucet, ToggleOff), (Pan, PickUp), (Table, Put)], all receptacle objects and "Pan", "Faucet" will the C' objects indicated on the map.
A.3 SEMANTIC SEARCH POLICY MODULE
The map from the previous subsection is passed into 7 layers of convolutional nets, each with kernel size 3 and stride 1. There is maxpooling between any two conv nets, and after the last layer, there is softmax over the 64 (8 8) categories, for each of the C, (73) channels.
At deployment/ validation, if the agent is currently searching for the cth object, then a search location sampled from the eth channel of the outputted 8 x 8 C. grid.
13

---

Published as a conference paper at ICLR 2022
7 Layors ConvNots
Conv
Pooling
Pooling
Conv
Softmax
A/M
Figure 9: Semantic search policy.
A.4 IMPACT OF GRID SIZE ON THE EFFECTIVENESS OF THE SEMANTIC SEARCH POLICY
While we chose . = 8, M the desirable choice of N may be different if practitioner attempts to transfer FILM to different scenes/ tasks. While "too fine semantic policy will be hard to train due to sparseness of labels, a "too coarse" one will spread the mass of the distribution to widely.
N 30 for the size of the "coarse" cell of the semantic search policy,
Let us examine the "coarse" and "actual" ground truth distributions just in one direction (e.g. the horizontal direction). Let Fx (x), Cx (x) be the "actual" and "coarse" ground truth CDFs in the horizontal direction. Also, let L N If the goal object occurs "k" times in the horizontal direction, then,
sup Cxx()) -
A similar result holds in the vertical direction. The bound above suggests that if the goal object occurs more frequently (smaller }), then coarser L (larger ) is tolerable. On the other hand, the goal object occurs very infrequently (larger }.) then a coarse M (larger - 1/ will result in Fx and Cx becoming L (and in turn, N) based on the frequency of their goal objects, on average. Furthermore, a search policy with adaptive grid sizing should be explored as future work.
different in the worst case. Thus, it desirable that practitioners choose
A.5 PSEUDOCODE FOR THE DETERMINISTIC POLICY
Following the discussion of Section 4.4, [(obji. actionn), ... (obje, actiong)] be the list of subtasks, where the current subtask is (obj,, action,). If obj, observed in the current semantic map, the closest obj, is selected as the goal to navigate; otherwise, the sample from the semantic search policy chosen the goal (Section 4.3). The agent then navigates towards the closest obj; via the Fast Marching Method (Sethian, 1996). Once the stop distance reached, the agent rotates 8 times to the left (at camera horizon 45, 90....) until obj, detected in egocentric vision. Once obj, is in the current frame, the agents decides to take action; if two criteria are met: whether obj, is in the "center" of the frame, or whether the minimum depth towards obj; in visibility distance of 1.5 meters). Otherwise, the agent "sidesteps" to keep obj; in the center frame or continue rotating to the left with horizon 0/45 until obj, seen within visibility distance. If the agent executes action; and fails, the agent "moves backwards" and the map gets updated.
Below, we present pseudocode for the deterministic navigation/ interaction policy. We first present explanations of some terms.
"visible" means that an object is in the current RGB frame, and minimum (predicted) depth from the agent to it is less than or equal to 1.5 meters (which is set by ALFRED).
"FMM" is Fast Marching Method (Sethian, 1996).
We assume that a new RGB frame is given as time_step +- time_step +
MoveBehind, SideStep, RotateBack are not actions in ALFRED; they defined by
MoveBehind RotateRight, MoveAhead, RotateLeft SideStep RotateRigh//Let, MoveAhead, RotateLeft/Right RotateBack RighteRight, RotateRight
14

---

Published as conference paper at ICLR 2022
Algorithm Navigation/ interaction algorithm in episode
1: Input: List of goal tuples [(ob]i. action1), (objk. actions)]
Output: Task Success True/False
4: timestep
5: goal pointer x-1
6: Sample from the semantic search policy
7: exeeute_interaction <- False
8: stop <- False
9: subtask_ssuccess &- False
10: move-pointer f-
11: task.success + False
12:
obji +- -bbjgont pointeri action, f- acliottoon pointer
14:
15:
while goal pointer S k do
16:
while timestep < 1000 do
17:
update semantic map
18:
19:
if stop then www.cute..iiiicccc exzecute.interactionn then
20:
21:
Execute action, if actioon, done successfully then
22:
23:
subtask-success
True
24:
else
25:
if objj visible in current frame and objs in the center of the frame then ezecute_inte iiterutiintercctionnn False
26:
27:
Execute LookDown 0
b void action
28:
else
29:
if previous action was OpenObject or CloseObject and not subtask_success success then Execute MoveBehind
30:
31:
else previous action was PutObject and not subtask_succees then
32:
Re-dilate g the semantic map
33:
Execute RotateBack
34:
else obje visible but in center of the frame then Execute SideStep
35:
36:
else
Rotate with camera horizons 0", untunti obj, is visible
37:
if more poinder < then Execute RotateLeft else
38:
39:
40:
if mooeueporr "move_pointer = then Execute LookDown 45"
41:
42:
Execute Rotatel.eft
43:
move.pointer f- move-pointer + (mod 8)
44:
45:
else
46:
not (obj, found) then Exccute one of (Rotatel.eft RotateRight, MoveAhead) with FMM to
47:
48:
else
49:
g t- closest obj: the semantic map
50:
while distance g > 0.65 meters do Execute of (RotateLeft, RotateRight, MoveAhead) with FMM to
51:
52:
if distance to < 0.65 meters then stop +- True timestep +- timestep +
53:
54:
55:
56:
if timestep = 0 (mod 25) then Sample new from the semantic search policy
57:
58:
59:
if subtask_succes.. then
60:
goal- pointer + goal-poonnee pointer
61:
obj: <- objgoal.pointer. action: move_pointer +
62:
63:
erecule interaction <- False
64:
stop + False
65:
subtask_successs +- False
66:
Sample new g from the semantic search policy break
67:
68:
69:
goal pointer = ++1 then
70:
task_success <- True
15

---

Published as a conference paper at ICLR 2022
A.6 MORE EXPLANATIONS ON TABLE 3
Table shows common error modes and the percentage they take out of all failed episodes, with regards SR. More specifically, it is showing the distribution of episodes into exactly one error mode, out of the 79.9% of all "Val Unseen" episodes that have failed (the episodes not in the 20.10% 10% of Table 2). The common error modes are failures in (1) locating the subgoal object (due to the small field of view, imperfect segmentation, ineffective exploration), (2) locating the subgoal object because is in a closed receptacle (cabinet, drawer, etc), (3) interaction (due to object being too far or not in field of view, bad segmentation mask), (4) navigation (collisions), (5) correctly processing language instructions, (6) others, such as the deterministic policy repeating a loop of actions from depth/ segmentation failures and 10 failed actions accruing from mixture of different errors. These errors occur in the order of (5), (1)/ (2), (3), (4) in an episode, since the LP module operates in the beginning and the object has to be first localized to be interacted with, etc. If an episode ended with errors in multiple categories, it was classified as an example of an "carlier" error in making Table 3. For example, if the language processing module made error and later there were also 10 collisions, this episode shown as a case of error (5) in Table 3.
A.7 ASSIGNMENTS OF ROOMS INTO "LARGE" AND "SMALL" IN VALID UNSEEN
There are 4 distinct scenes in Valid Unseen (one kitchen scene, one living room, one bed room, one bathroom). The kitchen (Large) has a significantly larger area than all the others (Small).
A.8 PROTOCOLS FOR REPRODUCING THE SEMANTIC POLICY
The primary result in Table 1 is from architecture tuning of the language processing, the semantic mapping, and the semantic search policy modules on the development data (validation unseen). Reviewers correctly noted that it is possible random seeds will also effect performance so the model was retrained four additional times and test results reported here. Since components of the language processing and the semantic mapping module were trained from pre-trained weights, we report the performance of FILM with semantic search policy trained from different seeds.
The improvement by the semantic policy as shown in Table is reproducible across multiple seeds. Table 8 shows results on Tests Unseen with semantic policy trained with different start- ing seeds (where SEED 1 denotes that the policy was trained with torch. wwm.maalaa.........mm... _seed (1) ) With learning rate of 0.001 and evaluation of every 50 steps, the model with the lowest test loss subject to train loss < 0.62 was chosen. The exact code and commands can be found here: https:/githuu.cco//yyeoommm/ Mittoonn the comantic-policy
Table 8: Results of FILM reproduced across different starting seeds of the semantic policy. The + error bar in the AvG. row denotes the sample variance.
Method
Tests Unseen
PLWGC
GC
PLWSR
SR
Low-level High-level Instructions
TABLE 1
15.06
36.37
10.55
26.49
SEED
15.12
38.55
11.34
27.86
SEED
13.82
36.58
10.13
25.96
SEED
10.47
37.12
14.05
25.64
SEED
14.22
37.37
10.69
26.62
AVG.
13.74
37.20
11.352
26,51 d: 0.58
High-level Instruction Only
TABLE 1
13.13
34.75
9.67
24.46
SEED
14.05
36.75
10.47
25.51
SEED
12.60
34.59
9.07
23.48
SEED
12.86
35.02
9.23
23.68
SEED 4
13.61
36.10
10.10
25.18
AVG.
13.25
35.44
9.71
24.87
0.64
A.9 A LANGUAGE PROCESSING MODULE WITHOUT THE TEMPLATE ASSUMPTION
The second paragraph of section 4.1 explains the template assumption, with the tasks belonging to one of the 7 types. For direct comparison with existing methods that do not take direct advantage of this assumption, we trained new Language Processing module that does not make use of templates
16

---

Published as a conference paper at ICLR 2022
but makes of the subtasks sequences annotations ALFRED provides." Fine-tuning a pre- trained BART (Lewis et al., 2020) model, we directly learned a mapping from a high-level instruction to sequence of subtasks (e.g. "Drop clean pan the table" - "(PickupObject, Pan), (PutObject, Sink),...") Without any assumption on the structure of the input and the output, this model takes a sequence of tokens input and outputs sequence of tokens. With the new LP module, obtained SR of 18.03% on valid unseen, which is a slight drop compared to our original 20.10% indicating that templates are only marginally helpful in performance.
For future research, we believe templates should be used instead of subtasks annotations, since they are much cheaper to obtain naturalistic settings. In this work, we created the templates (one for each type) by writing down an intuitive canonical set of interactions to successfully perform the task. To do so, we looked at just episodes in the training set and spent less than 20 minutes creating them; these cheaply obtained templates cover all 20,000 training episodes. Even to train an agent to perform more complex tasks, it more realistic to use templates than assume sub-task annotations.
On the other hand, our findings simultaneously suggest the need for better program synthesis method from instructions to subtask sequences, for general purpose instruction following not bound to certain "types" of instructions.
'Existing works(Blukis et al., 2021; Kim al., 2021; Zhang & Chai, 2021; Pashevich et al., 2021) subtask sequence annotations (or expert trajectories that contain the subtask annotations) as well.
17


Reviews:
[{"review_id": "fPGIhtTXLoe", "reviewer": "Reviewer_dBxN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n+ The proposed approach leads to impressive (SoTA) results on the ALFRED benchmark without using expert trajectories or low-level instructions.\n+ The analysis presented is very interesting. Specifically, the authors have done a commendable job in discussing the potential failure modes of the system. The analysis might prove to be potentially insightful for future researchers working on this problem.\n+ Barring a few missing details / inconsistencies (refer to the remainder of the review), the paper is easy to follow and the main ideas have been presented in a clear and coherent manner.\n\n**Weaknesses**\n+ While the authors have demonstrated strong results on the ALFRED benchmark with their approach, I feel that the overall idea of using modular components to tackle long-range, target-driven navigation is not new and has been explored sufficiently in prior work (https://arxiv.org/abs/1907.02022, https://arxiv.org/abs/2110.02207, https://arxiv.org/abs/2007.00643, https://arxiv.org/abs/2007.00643, https://openreview.net/pdf?id=HklXn1BKDH, https://arxiv.org/abs/1809.00786). Having said that, I didn't find any particularly novel insight from the perspective of using waypoint-guided navigation via modular methods (the fact that semantic/spatial memories help long range navigation is not very interesting at this point).\n+ While I do agree with the evidence for the claim that FILM generalizes well, however, there seems to be a huge gap between the performance of FILM and other competing approaches under the setting where low-level instructions are used in Tests-Seen (with FILM turning out to be inferior than the rest). This suggests that the proposed approach is not able to make use of the low-level instructions optimally. This is also evidenced from the trends seen in Tab. 2. Can the authors share their thoughts about this?\n+ Within the Language Processing module, the authors use the oracle sub-task decomposition template for each question type and fill in the blanks with the output from the argument predictor. This seems to be privileged information that would greatly help modular methods. the authors must clarify whether all competing methods make use of this information and if not, then does it still remain a valid comparison?\n+ The text (although overall well written), seems to be missing some critical details that might make a full comprehension of their approach problematic. Please see below for specific examples on missing information.\n\n**Some questions/clarifications/suggestions**\n+ Although not a critical issue at all (doesn't factor into the review), but I felt that the name of their approach FILM might get confused / overloaded with the visual reasoning model with the same name (https://arxiv.org/abs/1709.07871). Just thought that the authors should be aware of this as an FYI, if they are not already.\n+ While going through the Introductory sections of the paper, I felt that a concrete definition (in terms of inputs, outputs and assumptions) of the Embodied Instruction Following (EIF) task as defined on the ALFRED benchmark is missing. Without that, it feels like the authors are talking about EIF in the generic sense of the term. Within that context, it becomes a little confusing when a comparison with VLN is made (because, again, speaking in general terms, VLN is also an instance of EIF).  The text later clarifies the exact task definition in the subsequent sections, but I felt that the definition should have come sooner.\n+ The text refers to ALFRED as an environment at one place. I'd consider it as a benchmark (or, a dataset) rather than an environment.\n+ The description of the argument prediction sub-module of the LP module talks about 4 kinds of arguments: “obj”, “recep”, “mrecep” and “sliced”. Whereas, in the figures and in the later sections, there is mention of “parent” being one of the argument types in place of “recep”. The authors should clarify the list of possible argument types and fix the inconsistencies.\n+ The paper doesn't talk in detail about how the low-level instructions are used by their proposed modular approach. What parts of the existing pipeline are modified and how?\n+ What do the authors mean when they say that the task is deemed successful within a tolerance of \"10 failed actions”? Do these refer to the low-level actions (forward, left, right, …)? Or, are these high-level goals such as \"slice bread” or both? Might be useful for readers who are not aware of the specifics of the ALFRED benchmark evaluation.\n+ Is the aggregation over time steps in the Semantic Map Module done using something like simple averaging? Or, via some sequence models?\n+ It seems like the number of semantic categories whose information is being saved at each of the MxM metric locations on the top-down map is different for different episodes (C+num of sub-goal objects). How is this handled?\n+ Is the agent working with an egocentric or an allocentric map? The Figures seem to suggest an allocentric map. But the description of the semantic search policy states that the GT location for small objects is computed every 25 steps, which seems to suggest an egocentric map. The authors should consider making this explicit in the text.\n+ How is the overall framework keeping track of completed sub-tasks and deciding to advance to the next one?", "summary_of_the_paper": "The paper proposes a modular framework for solving instruction following tasks from the ALFRED benchmark. The approach uses a language module that takes the high-level instruction, predicts the instruction type and uses the oracle template for the instruction type to fill in the arguments (objects/receptacles) and generate a list of sub-tasks to be executed on those objects and receptacles. The agent builds and maintains a spatial+semantic map to keep track of state and uses it to plan the next waypoint (which object/receptacle to navigate to next) as its intermediate goal. Their approach demonstrates the ability to ground small objects on the top-down semantic map and yields SoTA performance on the ALFRED benchmark. The authors also present a comprehensive analysis of their ablations and failure modes.", "summary_of_the_review": "I am looking forward to clarifications on missing information from the authors during the discussion phase.  Overall, while I do agree that the proposed approach shows impressive results on the ALFRED benchmark and the paper has useful insights, however, the overall approach doesn't have much novelty to offer. Therefore, I feel that the paper is good/useful, but slightly below the acceptance threshold due to novelty concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YQCm2W_5kni", "reviewer": "Reviewer_qMNi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The presented method is currently (at the time of this review) the SOTA model based on the unseen test set split performance. This is the main strength I am basing my score on.\n- The experimental setup presents a thorough set of ablations and error analysis (please see below for some questions).\n- The semantic search policy, to my understanding, improves over the closely related work of Blukis et al. by replacing their random search policy.\n\n**Weaknesses**\n\n- In my perception the presented model is very close in structure to that of Blukis et al. 2021, with the main difference being that this model does not have a high-level controller and instead leverages domain knowledge to predict a templated high level plan. I believe assuming to know the task type taxonomy is a strong assumption to make and one that makes me concerned about the method's ability to generalize to other domains (particularly those for which such a taxonomy would not be made available). If accepted, I would encourage future versions of the paper to explicitly mention this set of assumptions. This point is the main weakness I am basing my score on.\n- Minor weakness, but I felt like some of the figures were a little difficult to parse and certain experimental details appear to be missing (see below).\n- Minor: The experiments presented in Table 6 are used to claim that the semantic search module helps particularly in larger scenes. However, the unseen validation set consists of only 4 scenes, each of a different category of room (kitchen, bathroom, living room, bedroom). I'm concerned that there may be a confounding factor in terms of the semantics of different rooms, and I'm wondering if any additional analysis was conducted to disentangle the effects of room size vs. that of room semantic category?\n\n**Suggestions**\n\n- Title: FILM is an acronym already in use by Perez et al. 2018 (FiLM: Visual Reasoning with a General Conditioning Layer), which may confuse readers. Just wanted to point this out for the author's consideration.\n- Abstract: \"This requires the use of expert trajectories and low-level language instructions\" — this statement feels a little strong to me and I would suggest qualifying it since not all existing methods require low-level language instructions.\n- Embodied Instruction Following: Very minor comment, but I have not seen this acronym be used before and I'm not sure if categorizing the task like this may confuse readers. Have other works used this term?\n- Figure 1: (1) I found the red dots a bit confusing — it's not clear to me in which direction the agent is moving due to lack of arrows and them persisting across timesteps. (2) It took a while for me to understand what the labels on the semantic map mapped to, perhaps color coding them to match the semantic segmentation would make it clearer. (3) Why is the blue square so regular (ie a perfect square), whereas the rest of the semantic segmentation map is not?\n- Page 6: \"HLSM\" was not previously defined in the text.\n- Table 1: I would suggest stating what the bold, blue, and underlined numbers mean to make it easier to parse by readers.\n- Figure 6 seems appears to be a little too close to Tables 6 and 7 — at first glance it appeared that \"Instruction: Put a large clean knife on the counter\" was supposed to be read with Table 6.\n- \n\n**Questions**\n\n- How do you determine what objects are \"small\" and which are \"large\"? I was also confused by the statement in Section 2 comparing the semantic exploration policy to that of Chaplot et al.: \"employs a semantic exploration policy, our and their semantic policies serve fundamentally...\"\n- If the model was run across multiple seeds, I'm wondering what the standard deviations were for the presented results?\n- How were failure cases classified into each of the 6 error modes? I was very interested by this experiment, and I think the presentation could be made stronger if this is specified in the text.", "summary_of_the_paper": "This paper presents FILM (Following Instructions in Language with Modular methods), a model built for the ALFRED dataset. FILM consists of: \n\n1) A language processing module which uses a BERT backbone to (a) predict 1 out of 7 task types in the ALFRED ontology and (b) predicts slot categories for populating a template (1 pre-defined per task). \n\n2) A semantic mapping module which uses segmentation and depth prediction to populate a 2D spatial map of object class locations. \n\n3) A semantic search policy which uses a CNN to map from the semantic map to subgoal object locations. \n\n4) A deterministic policy which is used to navigate to either (a) a subgoal object location if it exists in the semantic map or (b) a sampled location from the search policy if not, and performs a predefined object interaction based on the current subgoal from the task template.\n\nIn my perception the main technical contribution of the work comes in the form of the language processing module, which provides much of the structure used for planning actions by the agent. \n\nThe paper also contributes a semantic search policy for reasoning over a semantic map to predict subgoal object locations (instead of randomly picking a new search location). \n\nExperimentally, the paper presents: \n\n1) SOTA results on the task when only the high-level instruction is given (low-level instructions witheld). \n\n2) Ablations showing a bottleneck from errors in the perceptual side and marginal gains from adding low-level language instructions. \n\n3) A taxonomy and distribution of error modes for the presented model, showing most errors come from either the language prediction, goal object location prediction, or goal objects being in closed receptacles (which the deterministic policy does not open). \n\n4)  An analysis showing that task type length does not correlate with performance.", "summary_of_the_review": "I am leaning in favor of acceptance because the method presented achieves SOTA performance on the ALFRED task, showing how domain knowledge can be leveraged for improved performance. \n\nHowever, I do not feel strongly about this stance because the performance boost appears to be achieved by baking in domain knowledge that may keep the method from being generalizable to other domains. \n\nDespite this concern, I think this work would still be of interest to the community since it may also serve to highlight the need for improvements in high-level planning/control in these types of visual language navigation tasks. Similarly, other components of the model are also hand-crafted and could serve to highlight ripe areas of work for the community (e.g. a low-level deterministic policy is used, which does not open receptacles). \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XCVkXzBlFQ3", "reviewer": "Reviewer_NHgG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Thank you\n\nThank you for your comments. After reading them and other reviews, I find myself convinced in part by the arguments. In particular this one on the templates. I see the authors commented about them. I think this account but an important part of the generalization. \n>\"FILM makes use of 7 highly specialized and ALFRED-specific task templates. This takes advantage of the templated nature of the tasks in the ALFRED environment, somewhat trivializes the language understanding task, and will not generalize to new tasks without new template engineering.\"\nHowever, it's fairly common in robotics tasks to have a series of high-level plans.\n\nI also became more concerned about novelty.\n\nI'm revising down my score. Thanks again.\n\n# Previous main review\nThe main contribution of the paper is the combination of the modules.\n1. Parsing the instruction. Purely based on NLP\n2. Exploring the map for building the global map and to locate objects. Based on a combination of SLAM and estimation of object presence.\n3. Interleaving the exploration with the selection of the next tasks/goal necessary for achieving the goal of the instructions.\n4. Using a hand-written policy for going to the next point of exploration.\n\nModule 2 is also a contribution per se, as it is designed to address the difficulty of finding small objects. Finally, modules 3 and 4 are contributions, not as isolated innovation but as they show their empirical power.\n\nBesides the impressive difference in performance, I am convinced about the argument of modularity. For instance, from natural language input, only the goal instruction is found to be necessary. Improvement in NL processing would independently increase the quality of the approach. Such separation might prevent interpreting the instructions given what was found in the environment. That's not relevant for ALFRED as a limited kind of object appears in all the domains.\n\nIn my opinion, the paper has two main potential weak aspects, but I don't think they are ground for rejection.\n- The granularity of the semantic search policy might be hard to control. However, I am convinced by the discussion in appendix A.4\n- The deterministic policy is not just using FMM. The pseudo-code in the appendix also includes a policy for exploring the current location.\n\nI have the following question I'd like the authors to answer:\n\n- I'm confused about how the arguments are used. Section 4.1 explains the 4 types. Later on, \"recep\" is easier to detect while \"obj\" is harder. \n\t- Are these four kind of arguments treated differently by the semantic mapping module? \n\t- C is the number of \"objects\", but I am not sure if that refers to the 1st kind of argument or objects in general. \n\t- On the other hand, the policy is trained only with the receptacle. What about the \"sliced\" and \"mrecep\"?\n\t- Please provide a summary of how the different kind of arguments are used across the whole paper.\n\t- Please indicate where such clarification would be added in the paper.\n\nHere are other comments and questions that I don't think deserve a direct answer during the rebuttal:\n- Table 3: is this regarding the overall performance shown in the 1st row of table 4? Are these errors with respect to SR or GC? I think it'd be more illuminating to present this table in absolute numbers. One option would be to add the row of no errors so that the columns add up to 100.\n- page 6. Evaluation Splits. Would you please remind the reader the size of those sets? That would help to evaluate the significance of the metrics.\n- Table 2: It's not clear that \"base method\" is FILM with semantic search. I only confirmed it because \"base method\" is the same as \"overall\" in table 4.\n\t- Page 8: \"Thus, we use FILM with semantic search as the \"base method\" (default) for all experiments/ ablations\". Would you please say this before introducing Table 2?\n- page 8: \"Second, the number of the subtasks entailed with the task type does not strongly correlate with performance.\" This is not surprising since the templates make the length less important. I think it'd be better to mention the effective number of combinations or suggest that as real proxy for difficulty.", "summary_of_the_paper": "The paper presents a modular approach for solving Vision Language Navigation tasks and test it in ALFRED, a recent benchmark for that task. Compared with other benchmarks, ALFRED features a higher number of actions, objects, and an agent operating with an egoistic view. Solving the task requires multiple high-level actions, combined with low-level actions and exploration. As suggested by the authors of ALFRED, the benchmark requires exploiting the hierarchical structure, reusing skills in multiple contexts, and structured reasoning. The submission uses precisely all these aspects to achieve a significant improvement over the next entry in the ALFRED leaderboard.", "summary_of_the_review": "In summary, the paper proposes a modular approach for Vision Language Navigation tasks. The assumptions are heavily customized for ALFRED, but some general lessons can be used to other domains.\n\nThe most interesting contribution was the level of abstraction of the semantic map and policy that allows separating the exploration and exploitation from the natural language instructions and from acting given the current situation and information in the map. Furthermore, given that actions are related to objects, and the agent is located in an environment, it makes sense to have a predictable policy, allowing to adjust independently the issues related to the map.\n\nSo, I propose acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HrWDX126yN9", "reviewer": "Reviewer_aTek", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a modular system (FILM) for egocentric instruction following in the ALFRED environment. The system does not require expert trajectories and can operate without low-level instruction sequences. However, it does make use of templated mappings from 7 high-level goal types to low-level instruction sequences, and these are quite specific to the tasks in ALFRED. As well as presenting a novel language understanding module that uses engineered instruction-sequence templates, the paper introduces a new semantic search strategy that identifies potential locations for each of the small object sub-goals, based on large receptacle object locations. The semantic map building module, and within sub-goal deterministic policy are motivated by previous work. However their incorporation into the FILM system is novel.\n\nExperiments show that FILM generalizes well from environments that were seen during training to those that weren't. This generalization is impressive and in contrast to previous systems, which all suffer from a significant performance drop when evaluated on unseen environments. FILM is evaluated both in the setting where low-level instructions sequences are proivided, and when only high-level goals are available. FILM performs better than previous work on unseen environments across the board; worse than previous work on with low-level instructions on seen instructions; and marginally better than previous work without low-level instructions on seen environments. However, the comparison to other work that doesn't use low-level instructions is a bit murky. FILM has access to a small set of task templates that somewhat trivialize the high-level to low-level mapping and I think this needs to be discussed more.\n\n**Strengths**\n- FILM generalizes much better to unseen rooms than previous work.\n- Semantic search policy improves accuracy, efficiency, and is particularly effective in large rooms and with difficult to detect sub-goals. This bodes well for even more complex environments.\n- NLP module mapping high-level natural-language instructions to low-level instruction sequences is very effective and maintains most of the performance of the system that has access to low-level instructions. Outperforms other systems that don't have access to low-level instructions.\n\n**Weaknesses**\n- The comparison to other systems that don't have access to low-level instructions is a bit murky. FILM makes use of 7 highly specialized and ALFRED-specific task templates. This takes advantage of the templated nature of the tasks in the ALFRED environment, somewhat trivializes the language understanding task, and will not generalize to new tasks without new template engineering. I feel the approach is valid but the paper would be stronger with a more thorough discussion of the tradeoffs between sytems that require low-level instruction sequences; systems that are constrained to a predefined set of templated instruction sequences; and systems that learn the high-level to low-level mapping without constraints.\n", "summary_of_the_paper": "This paper presents a modular system (FILM) for egocentric instruction following in the ALFRED environment. The system does not require expert trajectories and can operate without low-level instruction sequences. The system has the following components:\n\n1. A language processing module that maps high-level instructions \"Drop a clean pan on the table\" to a low-level sub-task sequence. This module makes use of a number of BERT based classifiers that:\n\n   a.  classifiy the high level instruction into one of seven possible goal types, each of which is deterministically associated with a templated sub-task sequence\n\n   b. identify object IDs to fill in the slots in the template identified in (a)\n\n2. A semantic mapping module that populates a grid with binary classifications that indicate object presence; obstacle presence; and whether the location has been explored. This module is inspired by previous work on egocentric navigation.\n\n3. A novel semantic search policy that predicts the likely location of small sub-goal objects (bowl) on the basis of large receptacle objects (table). This is trained via supervised learning on sampled trajectories and it samples goals at a coarse tiem scale of every 25 steps.\n\n4. A deterministic policy that operates within each 25 step search goal period. This makes use of the Fast Marching Method and is inspired by previous work on this task.\n\nTogether, these modules make up a system that generalizes to unseen environments much better than previous work. The new semantic search policy provide small but consistent improvements over a version of FILM that has this module removed.\n\nFILM is evaluated in both the setting where low level instruction sequences are provided, and the setting where only the high-level goal is provided and FILM must map this goal to low-level instructions via module (1). In both settings, FILM significantly outpeforms previous work on unseen environments. When low-level instructions are provided, FILM does worse than some previous work on seen environments. When low-level instructions aren't provided, FILM does marginally better than previous work on seen environments.\n\nExperiments show that the semantic search policy increases path-weighted metrics proportionally more than absolute metrics, suggesting that this policy improves efficiency as well as overall success of the search. Targeted evaluations also show that this search policy is particularly useful in large rooms, where the agent cannot see many target objects, and for the \"Clean and Place\" task that involves a difficult to detect object (sink) contained in a simple to detect object (countertop).", "summary_of_the_review": "- Novel modular egocentric instruction following system with multiple technical contributions.\n- Generalization to unseen environments is impressive.\n- Dependence on task-specific instruction templates needs to be discussed in relation to previous work. Claim that this can be directly compared to other work that does not make use of low-level instructions is questionable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a modular system named FILM, for egocentric instruction execution task in the ALFRED environment, which uses structured representations that build a semantic map of the scene, perform exploration with a semantic search policy, to achieve the natural language goal. They achieve strong performance while avoiding both expert trajectories and low-level instructions. The reviewers all reasonably liked the paper (all reviewers gave 'marginally above the acceptance threshold' score) and appreciated the planner ideas + strong results; but many of them also had concerns about the use of templated mappings from 7 high-level goal types to low-level instruction sequences, and whether this will make the system specific to ALFRED. The authors did provide some new results in the response period to show that results drop without the templates but not by a large margin. Some reviewers also had concerns about the novelty of the work and said that the semantic map building module and sub-goal deterministic policy are motivated by previous work, but their incorporation into the FILM system is novel. Lastly, there was some concerns/debates on whether the system assumes/uses too much domain knowledge / task type taxonomy which might reduce the ability to generalize to other domains / data types, versus on the other hand the results may also serve to highlight the need for improvements in high-level planning/control in these types of visual language navigation tasks.
Decision: Accept (Poster)