Hierarchical Modular Framework for Long Horizon Instruction Following 
Authors: Suvaansh Bhambri
Byeonghwi Kim
Roozbeh Mottaghi
Jonghyun Choi
Venue: ICLR 2022 Submitted (2022)

Abstract: Robotic agents performing domestic chores using natural language directives re-quire to learn the complex task of navigating an environment and interacting with objects in it. To address such composite tasks, we propose a hierarchical modular approach to learn agents that navigate and manipulate objects in a  divide-and-conquer manner for the diverse nature of the entailing tasks. Specifically, our policy operates at three levels of hierarchy.  We first infer a sequence of subgoals to be executed based on language instructions by a high-level policy composition controller (PCC). We then discriminatively control the agent’s navigation by a master policy by alternating between navigation policy and various independent interaction policies. Finally, we infer manipulation actions with the corresponding object masks using the appropriate interaction policy. Our hierarchical agent, named HACR (Hierarchical Approach for Compositional Reasoning), generates a human interpretable and short sequence of sub-objectives, leading to efficient interaction with an environment, and achieves the state-of-the-art performance on the challenging ALFRED benchmark.

Under review as a conference paper at ICLR 2022
HIERARCHICAL POLICY LEARNING FOR LONG HORI- ZON INTERACTIVE INSTRUCTION FOLLOWING
Anonymous authors Paper under double-bling review
ABSTRACT
Robotic agents performing domestic chores using natural language directives re- quire to learn complex task of navigating an enviromment and interacting with objects in it. To address such composite tasks, we propose hierarchical modular approach to learn agents that navigate and manipulate objects in a divide-and- conquer manner for the diverse nature of the entailing tasks. Specifically, policy operates at three levels of hicrarchy. We first infer a sequence of subgoals to be executed based language instructions by high-level policy composition controller (PCC). We then discriminatively control the agent's navigation by a master policy by alternating between navigation policy and various independent interaction policies. Finally, we infer manipulation actions with the correspond- ing object masks using the appropriate interaction policy. Our hierarchical agent, named HACR (Hierarchical Approach for Compositional Reasoning), generates a human interpretable and short sequence of sub-objectives, leading efficient in- teraction with an environment, and achieves the state-of of-thh--rt performance on the challenging ALFRED benchmark.
Example
Polley Learning
Action Policy
RIGHT
AIEAA
PICKUP
RIGHT
AlEAD
OPEN
PUT
CLOSE
OPEN
PickUP
Crose
Example
Policy Composition
Policy Learning
Controller
Time
Navigate to Tomato
Master Policy
COOL
MANICLATE
Pickup
Interaction Policy
Cool Tomato Refrigarator
DOWN
CLOSE
Objj. Fridge
Figure 1: 'Flat policy learning' proposed 'Hierarchical policy learning'. The flat policy learning has been employed in prior arts, training an agent directly learn the low-level actions. In contrast, our hierarchical policy decomposes a long-horizon task into multiple subtasks and leverages the high-level abstract planning, which enables an agent to better address long morizooizon planning.
INTRODUCTION
For the awwited dream of building a robot to assist humans in daily life, we now witness rapid advances in various embodied Al tasks such as visual navigation (Anderson et al.] 2018 Chen al.

---

Under review as a conference paper at ICLR 2022
2019
Chaplot al. 2017 Krantz et al., 2020). object interaction (huu et al. 2017 Misra et al. 2017 and interactive reasoning (Das et al., 20122018 Gordon et al. 2018). Towards building such re- alistic robotic assistant, the agent should be capable of all of these tasks to address more complex problems. A typical approach for combining these abilities is to build a unified model (Shridhar) et al., 2020; Singh al.] (02T) to jointly perform different sub-tasks. For example, the type of reasoning necessary for navigation differs significantly from the reasoning required for object inter- action; the former focuses on inferring the free space and search strategy to reach target location while the latter requires detection of objects and inferring their distances and states (Singh et al.) 202T]. Meanwhile, human cognition process learns to divide a task into sub-objectives such as navigation or interaction, which enables humans to facilitate complex reasoning in a variety of cir- cumstances (Hayes-Roth Roth & Hayes- Roth 1979p. Inspired by those, we propose a hierarchical agent that learns and infers low-level policies for different sub-tasks. To evaluate our proposal in challeng- ing scenarios, we particularly focus on the task of interactive instruction following, where the goal is to execute a set of navigation and object interaction tasks specified by natural language. This task poses numerous challenges including long-term planning, partial observability, and irreversible state changes. To complete a task successfully, an agent needs navigate through hundreds of action quences and interact with various objects in novel environments (e.g., dining room, and kitchen etc.) which necessitates thorough comprehension of both visual scenes and natural language instructions.
Specifically, we propose to disentangle the task into high-level subgoals which are executed by the respective subgoal policies. Our agent is composed of (1) a policy composition controller (PCC) that specifies sub-policy execution sequence, (2) a master policy (MP) that specialises in navigation and provides a signal conduct interaction tasks, and (3) several interaction policies (IP), which execute autonomous low-level interactions for each subgoal type. This disentanglement results in easier to analyze subtasks with shorter horizons.
Inspired by visual navigation literature (Yang et al. 2018 Wortsman et al. 2019) that encodes target object information for navigation, we propose an object encoding module (OEM) that provides such target object information and serves as a navigation subgoal monitor. During navigation to locate an object, the agent faces uncertain obstacles that forces it to traverse in a loop (deadlock state). We additionally propose a loop escape module (LEM) that assist the agent in circumventing such obstacles, based on a stored history of prior visual observations, thus resulting in better navigation.
We summarize contributions follows;
We propose a hierarchical framework that decomposes the instruction following task into semantic subgoals and effectively addresses them with corresponding submodules.
We propose object encoding module (OEM) that encodes object information from natural lan- guage instructions for effective navigation and loop escape module (LEM) to escape deadlock state with a stochastic policy.
The proposed agent achieves otate-of-the-art performance on the ALFRED benchmark.
2 RELATED WORK
Prior works propose numerous task setups and benchmarks for developing an agent to complete complicated tasks given natural language directives, such agents trained to navigate (Li et al.) 2020 Xu et al.] 2021) or solve household tasks (Shridhar et al.) 2020). However, the vast major- ity of approaches for these tasks employ flat reasoning (Singh al. 2021 Nguyen et al. 2021 in which the agent decides on the low-level actions accessible while moving through the environ- ment (Gupta et al. 2017 Zhu et al., 2020). When the prior arts seek to define subtasks. they are frequently defined in a shallow manner with only two layers of hierarchy (Zhang & Chai, 202T Corona et al. 2020 Blukis et al., 2021 Andreas et al., 2017: Gordon et al. 2018 Yu et al., 2019 Das et al. 2019). Furthermore, these strategies are inefficient in terms of data requirements. Nat-
ural Language is subjective, and even a seemingly simple command can contain several unstated meanings. Because of this semantic gap, most present solutions involve either a large amount of labelled data or and---an- learning to map language to low-level actions. Compounding errors also impede performance for activities with a long and complex horizon (Xu et al., 2019). We in- troduce a system for representing and using hierarchical knowledge in order to better the control of embodied agents doing complex tasks stated in natural language directives. Furthermore, because
2

---

Under review as a conference paper at ICLR 2022
of the modular structure, our agent reasons and accomplishes tasks along longer paths than previous studies, spanning numerous subgoals.
The task of interactive instruction following necessitates not only navigation but also the interaction
with surrounding objects by following natural language directives. (Shridhar et al. 2020) proposed LNN-LSTM-bassee baseline agent with progress tracking (aa et al., 2019). (Singh et al., 2021) offer a modular strategy that factorises perception for mask prediction and policy for action predic- tion. (Nottingham et al., 202T) offer a modular system in which language encoding, visual state extraction, and action prediction are all trained independently in different modules. (Nguyen et al.) 2021] suggested system for predicting actions and masks corresponding to different instructions independently. They combine numerous visual inputs from navigable directions to provide larger field of view for improved navigation. (Zhang & Chail (02T) propose a Transformer--aasee erarchical agent whereas (Suglia et al., (021) present Transformer-based agent that uses object landmarks for navigation. (Pashevich et al., (02T) present a transformer-basee agent that uses a multimodal Transformer for exploiting the multiple input modalities. (Blukis et al. 2021) propose technique for successful high-level reasoning by reconstructing a 3D map for encode spatial seman- tie representation. Finally, modular policy with a two levels of hierarchy for instruction following have previously been proposed by (Corona et al. 2020) which results in poor performance on long- horizon task. In contrast, our policy operates three hierarchical levels exploiting the fact that navigation and interaction are semantically diverse activities that requires independent processing.
hi-
APPROACH
We observe that the visual information for navigation varies considerably over time while for object interaction it is largely stationary. As a consequence, the agent benefits from learning different policy modules for the two tasks, because the former needs to reason about temporal history and global environment information, whilst the latter requires to focus on local visual cues for mask prediction. Furthermore, the class imbalance between navigation and interaction atomic actions (i.e., navigation actions are far more frequent than the interaction actions) would bias a model towards more frequent navigation actions.
Inspired by these motivations, we design our modular architecture with three levels of hierarchy; (1) A high-level policy composition controller (PCC) that uses language instructions to generate sequence of sub-ohjectives, (2) master policy that specialises in navigation and determines when and where the agent is required to perform the interaction task, and (3) interaction policy that is a collection of subgoal policies that specialises in precise interaction tasks.
We first analyze each language instruction, and use the information determine the basic interaction policies required to perform the task. Depending on the underlying composition, the control of the agent is alternated between (1) master policy and (2) various specialised policies for object interaction. Importantly, all interaction policies are compositional and independent, which allows formulating an instance specific policy execution sequence. In particular, we learn seven interaction policies, each of which specialises in different sub-objective and can be integrated in precise order to accomplish long-horizon planning tasks. We illustrate the overview of our proposed agent in Figure 2
3.1 POLICY COMPOSITION CONTROLLER
For the task of interactive instruction following, the trajectories can be divided into several mean- ingful subgoals and detailed instructions are provided for each subgoal. Due to the requirement of long horizon planning for the task, we believe that it should be beneficial to first generate a high- level subgoal layout for the given task and then tackle each subgoal individually. To this end, we propose policy composition controller (PCC), shown in Figure 2 which generates a sequence of subgoals S = { (where s; belongs to the set of subgaols defined by benchmark) for each 'step- by-step' instructions corresponding to the given task. The PCC's predictions correlate to semantic subgoals, subjecting the agent's 's logic to observation ( ('What is the agent attempting to accomplish right now?"), which enables
monitor the progress of task completion by the agent. Specifically, we first encode the language instructions with a Bi-LSTM. followed by a self-attention module. The encoded step-by-step language instructions &; are used as input for the PCC to generate the subgoal
3

---

Under review as a conference paper at ICLR 2022
Language Instrucioon
Pulcy Compositin Controler
Maxter Palicy
Predicted Navigation Action
Olipal Secading
tpliion Palley
Leep Essope
Rg R2-22
Tun int pnd
Paxtle
CHl
Op. 1,-
HADE thes porsit the microoses
HisDDEBCY
Interaction Policy,
Predicted Otject Mask
beasadio Packey
Permerkos
Percsption
Iranedon Polen
Pebryy Morsto
Predicted Interattion Action
+
O
dep
12
Figure 2: Model Architecture. The model architecture comprises of three main components (1) policy com- position controller for defining the policy structure based on the given instructions, (2) a master policy (MP) responsible for navigation, and (3) set of interaction policies (IP) specialising in the manipulation tasks. I denotes an RGB frame from an explorable direction, d E [0, D]. at the time step, where d = 0 indicates the egocentric direction. We encode T3' using pretrained ResNet and acquire a visual feature, v/ & denotes each step-by-step instruction. lr. fm.m denotes the encoded instruction for the 'interactive perception module' and "action prediction module' respectively. İr:T+I,- denotes the encoded 'subtask' instruction described in Sec. 3.2.1 T refers the index of the current subgoal. In master policy, OEM outputs object encoding, oe, using I, 'VL-Ground" uses dynamic filters to capture the correspondence between visual and language features and outputs attended visual features, 65""" and 0000
sequences. The agent must complete these subgoals in the specified order to accomplish the goal task. Formally, for each language instruction ž;, the PCC defines a distribution over the subgoal action space as:
s; = arg
where & E [1, N Neubgoats)
(1)
where FC1 denotes single layer perceptron, and Naubgoats denotes the number of subgoals i.e. 7. We train the PCC module in supervised manner with the associated subgoal labels, On the ALFRED validation fold, the controller achieves 98.5% accuracy. Kindly refer Sec. A.3 for more details about these subgoals.
3.2 MASTER POLICY
We observed that the agent does not require information about the interactions must perform for the given task when navigating in an environment. We require the agent to learn how to move from one interaction location to another with the help of language instruction. Inspired by these observa- tions, we contend to use independent module that can perform navigation while simultaneously marking the locations for object interaction along the way. For the task, we propose a master pol- icy, which is illustrated by the upper-right block in Figure 2] for generating the navigational action sequence based on the multi-modas inputs. Let Ay denote the set of primitive navigation actions Moveehead, RotateRight, Rotatel.eft, LookUp, Look The master policy learns navigate in the environment by learning a distribution over AA <MANIPULATES where <MANIPULATE the abstract token we introduced for the agent to signify when to move control the next level of hierarchy, i.e., the interaction policies, for completing manipulation sub-tasks. The master policy comprises of three basic modules: (1) object encoding module that provides information about (2) navigation policy that outputs the navigation action sequence based on the modal input for traversing the environment. (3) loop escape module that helps the agent in escaping deadlock states.
the object the agent needs to locate for interaction.
3.2.1 SUBTASK LANGUAGE COMBINATION
The language instructions for the interactive instruction following task can be divided into two types: a) Navigation and b) Interaction. The agent needs to navigate to the necessary location and then perform interaction tasks, because of which the two types of instructions occur in pairs. To describe the complete task, the combination is repeated several times with varying locations and interaction
4

---

Under review as a conference paper at ICLR 2022
subgoals. For the better utilisation of the language input, we propose new way of encoding step- by-step instructions for the navigation task.
Specifically, we define subtask instruction combination of 'navigation to discover the relevant object' and the "corresponding interactions'. For instance, in the subtask Turn around and walk forward to the garbage bin on the floor by the TV. Pick up the blue credit card on the TV stand., the agent needs to interact with the credit card, which is a crucial information for the agent and also serves as navigational criterion, i.e., the agent should stop if it encounters the credit card in close vicinity. We observed that this information is missing in the first navigation command, but is present in the second interaction instruction which works as a motivation for the proposed combination. The navigation instruction provides the low-level information about the path to be traversed and the interaction instruction provides information about the object the agent needs to locate for appropriate interaction. We encode these subtasks using an instruction encoder comprised of a Bi-LSTM followed by self attention module. (T:T+1, refers to the encoded feature for the combined subtask instruction corresponding to the navigation subgoal T.
3.2.2 OBJECT ENCODING MODULE
The task of instruction following can be broken down into two components, according our ob- servations: 1) locating the required object, and 2) interacting with the object. We propose a module that takes as input the subtask language instruction (T:T+ I.n. and outputs an encoding for the object the agent must locate for interaction. This guides the agent's navigation, and we recommend using it as a subgoal monitor to signify the completion of the current navigation subgoal.
The master policy learns to output an abstract token <MANIPULATE> along its navigation route, which passes control a pretrained object detector (Mask RCNN) that validates whether the object provided by the OEM is present in the current frame or not. If the agent discovers the item, switches to the appropriate interaction policy; otherwise, it continues with the navigation. The object encoder is composed of Bi-LSTM, followed by a two layer perceptron which outputs the object class. For this object, the master policy utilises a learnable 100-dimensional embedding as input, demonstrated in Eq.
Navigation Subgoal Monitor:
3.2.3 NAVIGATION POLICY
The navigation policy II, receives the processed modal data as input and produces a sequence of actions as output. The architecture is based on the action prediction module of (Singh al. 2021). It uses visual features, subtask instruction features (Sec. (32.1], and the embedding of the preceding time step action inputs. Moreover, the object encoding supplied by the object encoding module (Sec. 3.2.2) is provided as input in order to further enhance the navigation capabilities of the agent.
The subtask instruction encoding (T.T+1 provides low-level information relevant for navigation as well as the information about the object that the agent needs to interact with. This aids the agent in arriving at the correct location. To capture the relationship between the visual observation and language instructions, we dynamically generate filters based on the attended language features and convolve visual features with the filters, denoted by "VL-Ground" in Figure 2 To summarise, the LSTM hidden state h of the master policy decoder, LSTM,, is updated with four different features concatenated together as:
0: argmax((FC, ((.:TI,.. where k [1, Nobjects]
hr. LSTM,, (lopar; I at-1,n; 0z])
dt,n aFC. ( (rerr 1,5 oG h), where ls E [1, +1]
where open denotes the attended visual features for the surrounding view as explained in Sec. A.2 in appendix at time step t, 1, denotes the attended subtask language features, dr-1,n represents the action given by II, in the previous time step, and o, denotes the object encoding given by the object encoding module. In addition,
use the subgoal progress and overall progress monitors similar to (Shridhar et al. 2020) for training the navigation policy. Details about the surrounding views is given in Sec. A.2 in the Appendix.
5

---

Under review as a conference paper at ICLR 2022
Agunt's View
Agent's
(a) Deadlock State
(b) Loop Escape Module
Figure 3: Loop Escape Module. The objective of the agent at the current time step move target object (a garbage can). (a) and (b) show an example a deadlock state and the behavior loop escape module when finding the target object, respectively. Each dark-blue square denotes the position of the agent. denotes the larget object that the agent should navigate to. - denotes the view direction of the agent. The dashed o and - indicate that the target object is invisible to the agent due occlusion. a path taken in deadlock state. a deadlock state, the loop escape module cancels the current action that results in the deadlock state, x, and takes a random navigation action, +.
denotes actions taken by the agent
3.2.4 LOOP ESCAPE MODULE
While navigating through an environment, the agent prone to caught around unanticipated obstructions such as tables or chairs, which might cause navigation failure. To circumvent such obstructions during navigation, (Singh et al., 202I) introduced obstruction evasion (OE). However, in the scenario where the agent predicts repeated series of activities (i.c., a deadlock state), the agent may fail to navigate even without such barriers, resulting limited exploration. The OE mechanism is unable to handle such challenges since repeated sequences of visual observations might not be recognised as obstacles based on visual similarity in consecutive frames. To address the problem, propose loop escape module, which allows for improved exploration inference time by avoiding deadlock conditions. We use an external memory to memorise the history of visual observations, inspired by (Du al. 2020). The agent identifies deadlock state if any exists that satisfies Equation] given a history of visual observations, Uo.r, up to the present time step, t.
Vw [].
Ue'+ Ut-w--
where W is a hyperparameter that indicates the length of the repeated actions. In the event of deadlock, the stochastic policy executes random navigation action assist the agent in escaping.
Figure3deppine an example of a deadlock state and the behaviour of loop escape module in the case when the agent unable find target object (a garbage can) due to occlusion (by the counter- top). As shown in Figure 3a... the agent in a deadlock state tries to revisit previously explored areas, which may limit exploration. Loop escape module assists the agent in escaping the impasse condition, thus finding the garbage can, depicted in F
3.3 INTERACTION POLICIES
To abstract a visual observation to consequent action, the agent requires global scene-level com- prehension of the visual observation whereas for localisation task, the agent needs to focus on both global well local object specific information. Inspired by (Singh et al. 20211, exploit separate streams for action prediction and object localization due to the contrasting nature of the two tasks, illustrated "Interaction Policy,' in Figure
The task require execution varied subgoals with different levels of complexity. For instance, Heat Subgoal might require interaction with either stove microwave whereas for Pickup Subgoal there is a variety of recepticles but the action sequence is simpler. To focus on individual sub- objectives, train module, II for each subgoal where k [1, Nsuhgoass. We observed that each interaction has its own properties and the navigation information history is irrelevant to

---

Under review as a conference paper at ICLR 2022
Input
Supervision
Validation
Test
Model
Vision
Language
Vision Language
Seen
Unseen
Seen
Unseen
RGB
Goal Instr.
Depth
Subgoal
Task
GC
Task
GC
Tass
GC
Task
GC
Seq2Seq
Single
3.70
10.00
0.00
6.90
3.98
9.42
0.39
7.03
LAV
Single
12.70
23.40
13.35
23.21
6.38
17:27
FPP
Single
25.85
34.92
5.36
16.18
26.81
33.20
7.65
15.73
EMBERT
Box
X
37.44
44.62
5.73
15.91
31.77
39.27
7.52
16.33
E.T.
Single
X
X
33.78
42.48
3.17
13.12
28.77
36.47
5.04
15.01
E.T. + synth. data
Single
V
46.59
52.82
7.32
20.87
38.42
45.44
8.57
18.56
LWIT
Box
33.70
43.10
9.70
23.10
30.92
40.53
9.42
20.91
HITUT
Box
25.24
34.85
12.44
23.71
21.27
29.97
13.87
20.31
HLSM
Single
25.00
36.37
11.80
24.70
25.11
35.79
16.29
27.24
HACR (Ours)
Single
X
45.96
54.55
19.96
38.19
40.66
47.62
16.70
26.17
Table I: Task and Goal-Condition Success Rate. / denotes the use of the corresponding attribute by the given approach and X denotes the absence of the corresponding attribute. "Input" column represents the various input modalities used by the given approach. "Single" and "Box" under "RGB" column denotes that the visual features are either extracted from the entire image or bounding box predictions respectively. "Goal" and "Instr" columns indicates the use of high-level goal instruction and detailed step-by-step instructions by the given approach respectively. ""uppervisoo" column provides details about the presence or absence of additional supervision in training the corresponding approach apart from the primitive actions and object masks. "Depth" column denotes depth maps RGB images and "Subgoal" denotes the high-level subgoal actions. We provide the highest values corresponding to each fold and each metric in blue and bold.
the task that makes it necessary to isolated hidden state for each interaction subgoal. More details about the architecture and training can be found in See. A.1 in the Appendix.
4 RESULTS AND DISCUSSION
First, we carry out a quantitative analysis task success (Success Rate or SR) and goal condition success rates (Goal Condition GC), and summarize the findings with prior approaches in Table
We compare our method with following prior-arts on ALFRED benchmark, Seq2Seq (Shridhar
2020 LAV (Nottingham et al. 2021 FPP (Singh al. 2021 EmBERT (Suglia et al.
2021
E.T. Pashevich et al. 2021) LWIT Nguyen el al. 2021
HITUT (Zhang & Chail 202
and HLSM Blukis et 2021).
HACR outperforms the preceding approaches on all metrics, as indicated in the table. Its ability to perform in unknown situations evidenced by its increased success rate in unseen environments. For the validation fold, HACR outperforms the state-of-the-ae (Blukis et al. 2021) with an improve- ment of 83.84 % in seen task SR, and 69. 15.55 in unseen task SR. For the test fold, HACR shows an improvement of 61.92 % in seen SR, and 2.5 % in in unseen SR. It indicates that our approach enhances the agent's overall grasp of all sub-tasks and their connection needed to complete task successfully, We also discuss performance over individual subgoals in Table 6 in Appendix.
4.1 COMPARISON BETWEEN HIERARCHICAL AND FLAT POLICY AGENTS
We present the analysis of the learning process for the hierarchical and flat policy agents for seen and unseen environments. The performance of our hierarchical agent and the flat agent are com- pared quantitatively in relation to the number of iterations (expressed in terms of epochs), and the results are presented in Figure As shown, hierarchical policy gives major improvement over the flat policy. Its ability to perform in novel environments is evidenced by its increased success rate in unseen scenes. As depicted in Table 2 (#b), for seen and unseen task SR, the hierarchical agent out- performs the flat policy by 9.5% and 49.5%, respectively. In both seen and unseen 'Goal-Condition' SR, the hicrarchical approach outperforms, with improvements of 8.3% and 29.1%, 1,, respectively. The greater performance of the hierarchical approach on both overall task success rate and goal condition suggests its comprehension of both short-term subtasks and long-horizon whole tasks.
Also, the hierarchical agent approach convergences significantly sooner than the monolithic agent (25th epoch vs 37th epoch), as shown in Figure/al demonstrating the computation-effiiiene of our approach. Our policies go through two stages of training. We start by training the seven interaction
7

---

Under review as a conference paper at ICLR 2022
Seen Success Rate
Hierarchical Flat
150
20
Policy
Flat Policy
100
20
35
Traliinng Epoch
Unseen Success Rate
20
10-
Hicrarchical icy
Pick Place
Cool & Place
Stack & Place
Heat & Piace
Clean Place
Examine
Pick 2 Place
Flat Policy
20
30
35
Training Epoch
(a) Success (SR)
(b) Mcan episode lengths
Figure 4: Hicrarchical policy learns faster and more efficient action sequences. Plot (a) shows the learning curves (success rates vs epochs) of the hierarchical and flat policy agents for unseen and seen environments. Plot (b) presents the average length of episode traversed hicrarchical or flat policy for the task
types ALFRED (Shridhar et al. 2020). The flat policy denotes the NIH ablated agent (#b) Table 2
policies, which collectively take two epochs to converge. We included them in the computation and began the hierarchical curve from the third epoch, which is the first epoch of our master policy.
Arrepreeents the average length of successful trajectory traversed by the hierarchical or flat policy agent, contrasting the efficiency of each agent. The hierarchical agent consist of the master policy that is dedicated solely to navigation, giving it a significant advantage over the flat policy agent that learns everything using the same network parameters. It was observed that due the wide action space, the flat policy agent occasionally executes irrelevant interactions along the trajectory, which is not the case with HACR. The dedicated action sets for the master policy and interaction policies improve HACR by allowing the agent to avoid any unnecessary interactions while traversing to discover the desired object. The interaction policy modules also perform significantly better because they only master certain short-horizon tasks, which speeds up and simplifies the learning process. Table Appendix provides information on the subgoal performances for each module.
4.2 INTERPRETABLE SUBGOAL MODULES
One of the benefits of the proposed approach that it generates semantically meaningful subgoals. The generated subgoals are far more interpretable than low-level action sequence. For instance, the low-level put action might be associated to any of the subgoals such as Heat, Cool, or Put. The high- level semantics are reasoned about by the hierarchical agent, and the agent's intent is considerably clearer. If the agent attempts to place the object in a refrigerator, it is most likely cooling it; the receptacle is microwave, it is attempting to heat it; and if the receptacle is table, it is simply pickup subgoal. In contrast, the flat policy agent considers it as single atomic action regardless of the object and receptable. We present an example for more detail in Figure 7 appendix.
4.3 COMPARISON BETWEEN MODULAR AND MON-MODULAR POLICY AGENTS
In modular networks, the decision makinn process separated into numerous modules. Each mod- ule is designed to perform certain function and is put together in a structure that is unique to each trajectory instance. Because of their compositional nature, such networks with the help of spe- cialised modules often perform better in new environments than their flat counterparts (Hu et al. 2019 Blukis et al. (011))
#c). We present a quantitative comparison of interaction policies' modular structure (Table #c). For this experiment, we train a single policy module to learn all interaction tasks. The decoupled pipeline for action and mask prediction, well the rest of the settings, are preserved. The modular agent outperforms the non-modular agent by 2.30% and 7.08% seen and unseen task SR, respectively. It also performs significantly well in both seen and unseen 'Goal-Condition" criteria, with gains of
8

---

Under review as a conference paper at ICLR 2022
Components
Validation-Seen
Validation-
Modular interaction policy
Navigation interaction hier.
Object encoding module
Task
Goal-Cond.
Task
Goal-Cond.
54.55 (.39
19.96
(8.19(1.))
/
(6,93
42. 12.22
12.14(0.2)
28.43(3.1)
(b)
41.96(2.1)
60.39
13.35(0.2)
29.59/2.4)
/
13.38 (4.9
18.64
34.98(1.5)
(8.96
37. 37.45
(.32
(8.29
(c)
39.64 (1.25
9.70
22.34(1.3)
Table 2: Ablation study for contributions of HACR. For each metric, we report the task success rate. The absence of checkmark (V) denotes that the corresponding component is removed from HACR. Here, the 'Modular interaction policy' refers to the subgoal modules, 'Navigation interaction hier.' denotes the hierar- chy between navigation and interaction policy (Sec. D. and 'Object encoding module' introduced in 3.2.2
Details are presented in Sec. 4.1 4.3 and 4.4 We report average over 5 runs (with random seeds) with standard
deviations depicted in sub-scnpt parenthesis
2.19% and 9.17%, respectively. The greater performance of the modular policy in both task and goal-condition metrics highlights the benefits of modular structure in long-horizon planning tasks.
4.4 ABLATION STUDY
We conduct series of ablation analysis on the components of HACR and report the results in Table 2 to evaluate the significance of each module with empirical studies.
(a) Object encoding module (OEM). First, we investigate the ablation of Object Encoding Mod- ule. We remove the navigation subgoal monitor and train the navigation policy without object infor- mation. The agent can complete some objectives, but it lacks object information, which functions as stopping criterion, preventing it from navigating properly. Therefore, it is unable to completely comprehend the relationship between the step-by-step instruction and the visual trajectory, limit- ing the agent's capacity to explore and connect various interaction policies required for the task's completion, thus resulting in a huge performance drop.
(b) Navigation interaction hierarchy (NIH). Next, demonstrate the importance of hierarchy between navigation and interaction policies. For this ablation, we utilize the same network for learning the navigation and interaction action prediction. For target class prediction, preserve the IPM. To eliminate the benefit of the language distribution, we use the concatenation of all step- by-step instructions the language input and conduct action and mask prediction while leaving the other modules unaltered. The ablated model's task success rates drop dramatically, showing that it unable to effectively utilise the available inputs.
(d) and (e) Individual Components Next, we provide the individual performance of the two major components of our framework which brings the most empirical gain, OEM and NIH in row (d) and (e) respectively in the absence of other components. Even in the lack of hierarchy and modular structure, the agent (d) has significantly higher performance than (Singh et al., 202T), D) highlighting the relevance of target object information for navigation and the effectiveness of our OEM. The lower performance of agent (e) denotes that the hierarchical architecture requires the inclusion of other components for optimal performance. The agent's performance increases when these components are combined (#c vs (#d and #e)) indicating that the components are complimentary.
5 CONCLUSION
We address the problem of interactive instruction following. To effectively tackle the long horizon task, we propose Hierarchical Approach for Compositional Reasoning (HACR), that decomposes the task into high-level subgoals and accomplishes each subgoal with specialized submodule, We propose a policy composition controller (PCC) to disentangle the task into high-level subgoals. For improving the navigation performance, we propose an object encoding module (OEM) to explicitly encode target object information during internal state update. We also propose loop escape module to circumvent the deadlock states during navigation, for better exploration. Our agent outperforms prior arts by significant margins and achieves new ofate-of-the-art

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
The embodied AI research will expedite the deployment of AI systems to help humans by its ver- satility of adapting to a new environment out of the factory or research labs. As all embodied AI, however, would suffer from adversarially annotated data well as data bias in form of language instructions, which may cause ethnic, gender or biased gender issues, the proposed method would not be exception. Although the proposed method has intention allow such problematic cases, the method may be exposed to such threats. Relentless efforts should be made to develop mechanisms prevent such usage cases in order to make the robotics AI safer and enjoyable be used by humans.
REPRODUCIBILITY STATEMENT
We take the reproducibility of research very seriously and describe the algorithm in detail in Sec. and Sec. I. We also solemnly promise to release all codes, environment information, learned models, and complete task configurations in public repository.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-anddlanguage navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. ICML, 2017.
Michael Bain and Claude Sammut. framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi, Learn- ing to map natural language instructions to physical quadcopter control using simulated flight. arXiv:1910.09660 2019.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi A persistent spatial semantic representation for high-level natural language instruction execution. arXiv:2107.05612, 2021.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2017.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modular networks for compositional instruction following. arXiv:2010.12700 2020.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em bodied Question Answering. In CVPR, 2018.
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural Modular Con- trol for Embodied Question, Answering. 2019.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning- NeurlPS, 2019.
Heming Du Xin Yu, and Liang Zheng. Learning object relation graph and tentative policy for visual navigation. ECCV, 2020.
Justin Fu. Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv: 10. /248, 2017.
10

---

Under review as a conference paper at ICLR 2022
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. CVPR, 2018.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017.
B. Payes-Roth and F. Hayes-Roth. cognitive model of planning. In Cognitive Science, 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. NewrIPS, 29:4565- 4573, 2016.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrlll and Kate Saenko. Are you looking? grounding to multiple modalities in Aision-and-languee navigation. arXiv:1906.00347, 2019.
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI, arXiv:1712.05474, 2017.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Diseriminnatoraccoorcriii Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.0000 1809.0992,, 2018.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. arXiv:2004.02857, 2020.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In ACL, 2020.
Chih-Yat Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitorin navigation agent via auxiliary progress estimation. In ICLR, 2019.
Dipendra Misra, John Langford, and Yoav Artzi Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. Look wide and interpret twice: Improving performance on interactive instruction-following tasks, arXiv:2106. 100e22060.0096. 2021.
Kolby Nottingham, Litian Liang, Daeyun Shin, Charless C. Fowlkes, Roy Fox, and Sameer Singh. Modular framework for visuomotor language grounding. In Embodied Al Workshop @ CVPR, 2021.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. arXiv:2105.06453, 2021.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Factorizing perception and policy for interactive instruction following. In ICCV, 2021.
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Em- bodied bert: A transformer model for embodied, language-guided visual task completion. arXiv:2108.0 04927, 2021.
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning learn how to learn: Self-adaptive visual navigation using meta-learning. In CVPR, 2019.
Danfei Xu, Roberto Martin-Mattnn De-An Huang, Yuke Zhu, Silvio Savarese, and Fei-Fei Li. Re- gression planning networks. In NeurlPS, 2019.
11

---

Under review as a conference paper at ICLR 2022
Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In NAACL, 2021.
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. arXiv:1810.0654: 06543, 2018.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, and Dhruv Batra. Multi- target embodied question answering. In CVPR, 2019.
Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring, arXiv:2106.03427, 2021.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In CVPR, 2020.
Yuke Zhu, Daniel Gordon, Erie Kolve, Dieter Fox, Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In ICCV, 2017.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DETAILS FOR INTERACTION POLICY
Let Am denote the set primitive interaction actions {Open, Close, ToggleOn, ToggleOff, Pickup, Put}. The APM architecture consist of an LSTM decoder that takes input the attended language encoding IT.m corresponding the current manipulation subgoal, the egocentric visual features f.mm and the previous time step action aim and outputs a distribution over the manipulation action space Am <STOP>. The manipulation policy uses <STOP> token as an indicator for completion of the interaction task and shifts the control to the master policy for further task completion.
Formally:
ft. LSTM ((0rg)) Im ag (,m))
(4)
a1m = (FC Im.m am 1)) where lt E [1, + 1].
The IPM uses the same inputs as APM and outputs the object class that the agent needs to interact with at the current time step. Similar to (Singh et al.] [2021), we use a pretrained Mask-RCNN to obtain the interaction mask corresponding to the object class. For more details about the architecture, kindly refer to (Singh et al., 2021).
1 ppm t = LSTM, ((tto) (, ::: 1.w,,
(5)
Cr,5 argmaa(FFC, (108%0 :0000, aei (i)) where [1, Nobjects).
ir
We train our interaction policies in supervised manner using imitation learning from expert groundtruth trajectories. Specifically, each interaction policy is trained to minimize the cross en- tropy loss as:
Lint Σoim log Pm EGi log Po:
where Pm FCm (10,90 & mi a4-1,m; ( and P. FC [0090;1 (i,,v' 0 h htp")) denotes the probability distribution of action and class prediction, respectively, and ai.m and cf, denotes the corresponding ground truth actions and classes obtained from given expert trajectories, respectively.
A.2 SURROUNDING PERCEPTION
HACR investigates and perceives surrounding views improve the agent's perception and gain better comprehension of the environment. Let vf be a visual feature from each navigable direction, D), the time step, t, where represents the number of navigable directions and d the agent's egocentric direction. After gathering additional observations, e..D we acquire attended visual features, aran using the dynamic filters and concatenate all attended visual features as:
pan
[00 0
Using the attended visual features, we update the master policy decoder and predict masks and actions as described in Equation 2
A.3 EXPERIMENTAL DETAILS
Dataset and Metrics. For training and evaluation, we used the ALFRED benchmark (Shridhar) ct al.) 2020). This benchmark provides expert trajectories for the agents performing household tasks (e.g. Boil an egg) in simulated environment AI2-THOR (Kolve et al.] 2017). The dataset is divided into three parts; 'train', 'validation', and 'test' set. To evaluate the generalisation ability of embodied agent to novel environments, the benchmark further divides 'validation' and 'test' trajectories into seen and unseen splits Unseen comprises a set of rooms which are held out during training and scenes that exposed to the agent during the training are termed as
13

---

Under review as a conference paper at ICLR 2022
The benchmark provides high-level Goal Statement describing the final task and several low-level Step-by-Sten Instruction describing each subgoal that needs to be accomplished the given order for successful completion of goal task. The benchmark has 7 types of subgoals; GOTOLOCA- TION, PICKUPORJECT, PUTOBJECT, COOLOBJECT, HEATORJECT, CLEANOBJECT, SLICEOO- JECT, and TOGGLEORJECT. GOTOLOCCATION denotes navigation subgoal where the agent requires to locate target object for interaction. PICKUPOBJECT requires the agent to pick up an object. PuTOBJECT requires the agent put an object onto a receptacle. COOLOBJECT, HEATOBJECT, and CLEANOBJECT requires the agent to cool, heat, and clean an object while interacting with the required receptacle (e.g., Fridge, Microwave, SinkBasin, etc.). SLICEOBJECT requires the agent to slice an object using knife, resulting in sliced objects. TOGGLEOBJECT requires the agent to toggle on or off an object such as a floor lamp.
Following the common evaluation protocols (Blukis et al. 2021 we evaluate our agent on two metries. Success rate (SR) provides the percentage of tasks that had all successful goal conditions. Goal condition rate (GC) is the percentage of satisfied goal-conditions across all tasks.
Training and Evaluation. We begin by assessing the performance of each subgoal policy re- lation to its specific task. On the ALFRED validation we present the success rate for each subgoal in T To accomplish this, we employ the expert trajectory to guide the agent through the episode until it reaches the subgoal task. The agent then uses the task specific policy to infer based on the language directive and current visual frame. Following that, we train and evaluate the master policy's performance during independent behaviour cloning on expert trajectories, i.e. suming oracle subgoal-policies. With ground truth subgoal sequences, the master policy is able to generalise effectively and achieves 29.96% success rate (SR) on unseen validation data. Then we train the Object Encoding Module and the Policy Composition Controller in isolation and combine all the modules together described in Sec.
Implementation Details. Visual features are encoded with pretrained ResNet-18 (He et al.) 2016) after resizing input images to 224 x 224. To perceive surrounding views, we addition- ally gather visual observations from four navigable directions (i.e., ROTATELEFT, ROTATERIGHT, LOOKUP, and LOOKDOwN). We train each module of our approach independently using the Adam Optimizer with the initial learning rate of 10 We augment visual features by shuffling the chan- nel order of each image (Singh et al., (021) and applying predefined image augmentations (Cubuk al. a (019)). We set the coefficients for the subgoal and overall progress monitor objectives 0.2, and 0.3 respectively. The value of W for the loop escape module is set as 10 using grid search on validation set.
A.4 FURTHER ABLATION STUDY
A.4.1 NETWORK COMPONENTS ABLATION
Components
Validation-Seen
Validation-Unseed
Object-centric mask prediction
Factorised interaction policy
Data Augmentation
Task
Goal-Cond.
Task
Goal-Cond.
15.9
34.55
38.19(1.5) 19.96
(a)
10.18
15.51
14.91
34.79
(b)
/
27. 27.15
38.50
11.11
33.81
(c)
/
9.27
20.95
2.56
22.67
Table 3: Ablation study for design components of HACR. For each metric, report the success rate. Checkmark denotes that the corresponding component is used. Please refer to Sec. A.4. details about each component. We report average over 5 random runs with standard deviations in sub-script parenthesis
(a) Data augmentation (DA). We start by ablating the data augmentations (described in Sec. A.3 which lowers the performance across the board, highlighting its usefulness in training smarter agent by reducing imitation learning sample complexity. This results in a performance drop of 12.6% and 25.0% in the seen and unseen task success rate respectively.
14

---

Under review as a conference paper at ICLR 2022
(b) Factorised interaction policies (FIP). Then, under the interaction policies, replaced the decoupled pipeline with a unified pipeline (Singh et al. 2021 for both action sequence prediction and interaction mask generation. Due shared processing for mask and action modules, the ablated
model's performance drops substantially, showing its inability to effectively utilize the language information.
(c) Object-centric mask prediction (OCMP). Finally, we train the interaction policies without the use of object based mask prediction which uses a Mask-RCNN as described in Sec. A.3 Similar (Shridhar et al, 2020), we directly upsample the joint embedding using deconvolution layers to generate the interaction mask. We see significant decline in performance on both seen and unseen folds due to poor mask generating ability, emphasising the need of object information for consistently accurate mask formation.
Validation
Validation- Unseen
Input Ablations
Task
GC
Task
GC
All imputs (HACR)
45.96
54.55
19.96
38,19
Egocentric View Only
30.48
36.13
12.42
23.72
No Vision
0.61
6.87
0.24
8.25
No Language
3.65
10.66
1.21
7.78
Goal-Only
8.53
16.50
2.07
10.04
Single Navigation Instruction
12.19
23.71
w/o Loop Escape Module
45.43
54.18
19.88
37.98
Table 4: Input Ablations and LEM Ablation.
We ablate different inputs to our hierarchical network and report the task success rates and GC success rates. Kindly refer Sec. A.4.2 for further details
A.4.2 INPUT ABLATION
To further investigate our agent's vision and language bias, we ablate the inputs to our model in T
Egocentric View only. This ablation specifies the condition under which the agent is deprived of its surrounding views (Sec A.2) while navigating without affecting any other module. The agent continues to perform admirably and outperforms the HLSM (Blukis et al. (202T) on the validation fold by large margin in seen trajectories, while also providing significant improvement in task success rate in unseen environment. This demonstrates the significance of peripheral perceptions in navigating novel environment.
No Language. We discovered that when the agent receives only visual inputs without the language instructions, it is still capable of doing some tasks by memorising common sequences and target classes the 'seen' fold. but does not adapt to the unseen environment.
No Vision. This ablation demonstrates that the agent can still perform some subgoals, mostly navi- gation, by following the instructions to along few short trajectories, but not able complete goal tasks because it involves mask generation (object selection). which is entirely dependent on visual observation.
Goal-Only. We define the setting where both the navigation and interaction policies receive only the goal task information. It gives general idea of the task but doesn't contain any information about the tedious action combinations the agent needs to perform get the task done. This leads to a significant depletion in the task success rate.
Single Navigation Instruction. This is when the master policy only receives the navigation in- struction corresponding to the current navigational subgoal. The agent able to somewhat navigate but doesn't 't have the object information that acts the stopping criterion which hinders the agent's ability to perform interaction at the correct place in the environment. This demonstrates the benefit of our object-centric navigational approach.
15

---

Under review as a conference paper at ICLR 2022
A.5 ADDITIONAL RELATED WORKS
In this section we extend our realated works, to provide a detailed technical comparison with the prior arts involving hierarchical frameworks (Blukis et al., 2021 Zhang
Chai
202T).
Comparison with HLSM (Blukis et al. 2021). Here we compare the technical aspects of HACR and HLSM. Both the works uses action merarchy to decompose trajectory into high-level subgoal actions and subgoals into low-level action sequences. There are differences between the subgoals anticipated by HACR and HL.MM. HL.SM only employs interaction subgoals and considers naviga- tion to be part of the same subgoal in order to reach the place of interaction, whereas HACR includes an extra navigation subgoal 'GoTo(object)" in which the agent navigates the environment to discover the object of interest. HACR also contains the compound subgoals for Clean and Heat, , whereas HLSM would forecast the complete sequence of primary subgoals like Put, Pick, Toggle etc.
In terms of model architecture, HLSM defines the task layout by using hierarchical model with high-level and low-level controllers. The navigation and interaction components of a particular sub- goal are collaboratively worked on by the low-level controller. HACR, on the other hand, adopt three-level hierarchy with independent modules for each level, namely, PCC, which provides the high-level structure of the trajectory, MP, which is responsible for navigation, and IP, which is com- posed of modules specialised for each interaction job.
We believe that HLSM and HACR are complementary because HLSM uses depth supervision es- timate the 3D layout of the environment for detailed visual understanding while our HACR exploits high-level language instructions for action planning. The depth supervision could be complemen- tary to low-level instructions because the supervision of HLSM provides rich visual information by depth, which contains visual details, while the supervision of our HACR provides abstract language information for high level guidance for actions. We argue that it is non trivial to claim which super- vision is less or more. Using the Object Encoding Module, HACR also adds a navigational subgoal monitor that checks the availability of object in the scene before executing the interaction policy.
A possible reason that HACR shows strong performance in seen but less in unseen is insufficiency of data. Due to the limited data size for imitation learning of the given benchmark (note that despite its size being largest in the literature, it is an offline dataset, i.e., the training data are obtained from the simulator by single expert trajectory not using the simulator to learn various possible scenar- ios) and large language domain gap in seen/unseen environments, generalization is challenging. As the detailed supervision about the environment, e.g., depth information, may help generalization (HLSM), using both supervision for environments and detailed language would reduce the general- ization gap further.
Comparison with HiTuT (Zhang & Chai 2021). We also compare the technical aspects of HACR and HiTuT. HiTut also employs action hierarchy to decompose trajectory into high-level subgoal actions and subgoals into low-level action sequences like HACR and HLSM. Even though HiTlT divides the task into a hierarchy of high-level subgoals and low-level actions, they utilize a unified transformer model with multiple heads to predict the subgoals and actions. In contrast, HACR consists of independent modules for subgoal planning, navigation and varying interaction tasks. The modular design allows for separate processing via specialised modules, making the learning process simpler due to short-term reasoning and enables easy inclusion of additional interaction subgoals.
A.6 ADDITIONAL EXPERIMENTAL RESULTS
There are seven high-level task categories in the ALFRED benchmark (Shridhar al. 2020 Table presents the quantitative comparison of our approach with flat policy, a seq2seq model (Shridhar al. 2020j, and prior hierarchical baselines (Blukis et al. 2021 Zhang & Chail 2021 for each task type. Blukis et al. (202T) outperforms our agent on short- tasks like PPick& & Place" and Exammin."" The two most complex and longest-horizon task categories in ALFRED are 'Stack & Place" and 'Pick Two & Place.' On these task types HACR gives an unseen success rate of 13.8% and 22.2%, respectively, compared to HLSM's 1.8% and 12.3%, and outperforms all baselines in of the 7 task types in unseen SR. Overall, HACR outperforms all the baselines in terms of average seen and unseen success rates. This is inline with the detailed analysis in Sec. and which describes the effectiveness of our approach to long-horizon trajectories.
16

---

Under review as a conference paper at ICLR 2022
Seq2Seq
Flat policy Unseen
HiTuT
HLSM
HACR (Ours)
Task-type
Seen
Unseen
Seen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Pick & Place
7.0
0.0
57.2
5.2
35.9
26.0
42.3
28.0
62.7
8.0
Cool & Place
4.0
0.0
40.9
19.9
19.0
4.6
11.9
3.7
46.0
31.2
Stack & Place
0.9
0.0
23.2
10.3
12.2
7.3
8.7
1.8
24.3
13.8
Heat & Place
1.9
0.0
47.7
11.4
14.0
11.9
12.1
0.0
52.3
16.9
Clean Place
1.8
0.0
38.8
24.6
50.0
21.2
25.0
10.6
43.7
38.1
Examine
9.6
0.0
48.5
10.3
26.6
8.1
37.2
22.5
Pick Two & Place
0.8
0.0
35.0
14.9
17.7
12.4
35.5
12.3
39.5
22.2
Average
3.7
0.0
41.6
13.8
25.1
13.1
24.7
11.3
45.9
19.9
Table 5: Success rates
task types in ALFRED. 'Seq2Seg" refers the method by Shridhar al.
2020). 'Flat policy' refers to Sec. Sec. (b). Here HiTuT (Zhang Chai 202T] and HLSM (Blukis all [2021 denotes the prior arts involving hierarchical aproach. We indicate the highest values corresponding to each task type in blue and all numbers are in percentage. We evaluate the agents on the Validation set.
Table presents the performance of individual subgoal policies on corresponding subgoals. Our hierarchical agent relatively outperforms the flat policy by 50% and 21% average on unseen and seen validation set respectively. The modular nature of our interaction policies allows them to learn specific short horizon subgoal tasks illustrated Figure which reduces the complexity and makes the learning faster and easier.
Seq2Seq
Flat policy
HACR (Ours)
Task-type
Seen
Unseen
Seen
Unseen
Seen
Unseen
Pickup
32
21
66
39
82
68
Put
81
46
73
45
90
68
Cool
77
95
94
69
49
85
87
Clean
81
65
51
73
Slice
25
12
65
49
73
79
Toggle
100
50
79
30
97
53
Average
71
50
86
75
Table 6: Subgoal success rate. The highest values per fold and are shown in blue.
A.6.1 ABLATION STUDY ON VISUAL INPUT FOR POLICY COMPOSITION CONTROLLER
The impact of visual features on PCC is presented in the Table The agent does not gain from accessing visual information, as evidenced by the drop performance for PCC vision. We believe that this happens due to the 'causal misidentification' phenomenon (de Haan et al. 2019); when receiving more information (e.g., RGB observation), the agent could learn to exploit irrelevant information (e.g., chair in the room) to target task (e.g., Heat potato) rather than necessary information (e.g., Microwave and potato), leading to performance drop.
Validation-Seen GC
Validation-Unsen Task
Input Ablations
PCC
Task
GC
PCC w/o vision
98.5
45.96
54.55
19.96
38.19
PCC w/ vision
96.8
45.12
53.01
19.49
37.59
Table 7: Input Ablations for policy composition controller (PCC). "PCC w/o vision" denotes the original HACR agent, where as "PCC w/ vision" denotes the agent using additional visual input for Policy Composition Controller (PCC) The highest values per fold and task are shown in blue. The second column depicts the performance of PCC for predicting the subgoal sequence evaluated validation
A.6.2 ABLATION STUDY ON VISUAL INPUT FOR MASTER POLICY
The impact of visual features on MP presented in the Table g The MPP w/o vision demonstrates that the agent still complete few goal conditions, mostly the first navigation instruction, by
17

---

Under review as a conference paper at ICLR 2022
following the instructions to move along few short trajectories, but not able to complete goal tasks because involves interaction with objects in the environment i.c. localising objects, which is entirely dependent visual observation.
Validation-Seen
Validation-Unseen
Input Ablations
Task
GC
Task
GC
MP w/ vision
45.96
54.55
19.96
38.19
MP w/o vision
1.58
12.57
0.73
10.27
Table 8: Input Ablations for Master Policy (MP). "MP w/ vision" denotes the original HACR agent, where "MP w/o vision" denotes the agent without visual input for Master Policy (MP). The highest values per fold and task shown in blue.
A.6.3 SENSITIVITY ANALYSIS FOR LOOP ESCAPE MODULE (LEM) WINDOW (W)
The impact of the value of W for Loop Escape Module (LEM) is examined in Figure by conducting a grid search. We observe that our agent achieves the best performance with W 10, which indicates the efficacy of LEM. Note that the performance does not drastically decrease by other values of W, which indicates that it prevents the agent from performance degradation by other values of W.
21
Success Rate (SR)
20
18
16
0
5
10
15
20
25
Length of repeated action (W)
The sensitivity curve for the value of loop escape module (LEM), Y-axis denotes success rate for the unseen validation split.
Figure 5:
A.7 WHAT TO LEARN? VS. WHAT TO BAKE IN AS AN INDUCTIVE BIAS?
We agree that our method more or less 'classical' pipelined approach and has many carefully designed components as a set of inductive biases. We attribute our design to (1) lack of sufficient training data and (2) requirement of heavy computational cost. (1) Given our framework is the behavior cloning (BC) (Bain& & Sammut] 1995), though the benchmark we used is the largest in the literature, the rigid nature of the BC approach (i.e., any deviation from the expert trajectory over time is not allowed to learn behavior) requires vast amount of training examples (i.e., may be not sufficient to be the largest dataset the literature) to learn satisfactory model. (2) To overcome the rigidity of learning BC model, there many approaches proposed in the literature (Ho & Ermon 2016 Fu et al.) 2017). But they are mostly computationally expensive (Kostrikov et al. 2018). We aim the niche between both problems by designing well crafted components for better encoding the given dataset even in the rigid BC framework.
Similar to answer to the previous question, our inductive bias could be useful for this task not just for this benchmark by the analogy of convolutional neural networks for images. By the uni- versal approximation theorem, the 2D information, e.g., images, could be encoded by multi-layer
18

---

Under review as a conference paper at ICLR 2022
perceptron (MLP) with sufficient amount of data. Given that it is difficult to fathom the sufficient amount data to train the MLP, we have a very successful design choice for encoding 2D infor- mation; the patch wise encoding and convolutional operations for its processing. Now, with 1M images, e-g., the ImageNet- K, we can have a decently performing image classifier. We hope that designed components could he useful to inspire the future research in designing data driven model to address the Embodied AI task successfully. In sum, we pose our work as stepping stone for designing and learning a successful data driven model the future.
A.8 DOES SEPARATE NETWORKS INCREASE FADDWAEE//..ANNNGCCOPPTAAAAAA COST?
In this section, provide discussion about the hardware, training, and computational cost when learning and exploiting separate networks for each subgoal.
A.8.1 HARDWARE COST
Our model would increase the memory cost linearly by the number of subgoals (0.23GB per subgoal) because models for each policy should be loaded in the memory proportional to the number of subgoals. Considering the large sized memory in modern computer systems (e.g., usually larger than 8GB for PC and 6GB for high-end mobile phones), the additional memory cost may not be considered large, but it is certainly cost.
A.8.2 TRAINING COST
The training cost for each policy may marginally or may not increase because we divide task into multiple subgoals and learning the specialized model for each subgoal takes few epochs as the training set for each policy easier (i.e., sharing similar semantics) for training than the data for the flat model. In Figure (a) in the revision, we show that learning the specialized networks requires less training epochs compared to learning a unified network. Therefore, even though there are multiple subgoals to be learned, the total training cost would increase marginally (and even embarrassingly parallelizable) or even decrease thanks to the reduced training cost per each subgoal added to overhead of predicting subgoals.
A.8.3 COMPUTATIONAL COST
The computational cost may marginally increase even decrease because specialized networks for subgoals could infer each subgoal efficiently by reducing unnecessary exploration. Figure 4 (b) in the revision shows that the specialized networks require shorter mean episode lengths to accomplish tasks. With the efficient subgoal achievement, the overall computational cost at inference would be similar decrease, compared to the model with the flat policy learning.
19

---

Under review as a conference paper at ICLR 2022
1.00
Pickup
Put
0.75
Cool
Success Rate
Heat
0.50
Clean
Slice
Toggle
0.25
0.00
0.0
0.5
1.0
1.5
Epoch
Figure 6: Plot shows the learning curve (subgnal success epochs) for the subgoal policies de- scribed Sec.
GOAL: Heat and chill potato.
Flat Pollcy
INSTRUCTION NONS:
Move Drverd and fice to the right counter
Carry the patato rightc
Tali pulito tollom the riait
braids.
3
POLICY
Hierarchical Pollcy
Subrask
Objjctt
nicrowave on lift Dook potato the
Microwave
pataoo to Vidge Open the tidge and placo ahe patato suidoo
Fridge
Pollcy Composition Controller
Pickup
HEAT
Cool
Figure 7: Inference of the task HHeat and Chill a potato.' using hierarchical policy vs. using flat policy. The hierarchical agent decompose the instructions into subtasks each including navigation and an interaction subgoal. The PCC at the bottom outputs sequence of subgoals used for selecting the appropriate interaction policy. The object column demonstrates the output of the OEM. The master policy the subtask instruction. object encoding, and visual observation to infer the navigation action at every time step as shown at the top. It predicts the <MANIPULATH On completion of the interaction task, the control is shifted back to the navigation policy. The hidden state of the navigation policy is keeps continuous encoding of navigation history without any interference of the interaction policy whereas each interaction policy uses an independent hidden state. The flat policy shown at top, on the other hand shares the same parameters for learning both navigation and interaction subtasks. This disentanglement results in simpler and faster learning by enabling parallel computation for various policies.
token to shift the agent's control to the interaction policies along the way.
20

---

Under review as a conference paper at ICLR 2022
PICKUP REMOTE
Figure 8: Pickup Object interaction policy agent completes a subgoal task "Take the remote." an unseen environment.
PUT PLUNGER
Figure 9: Put Object interaction policy agent completes subgoal task "Put the plunger below the sink." in an unseen environment.
PUTOBIECT MUG
TOGGLEON FAUCET
TOGGLEOFF FAUCET
PICKUPOBIECT MUG
Figure 10: Clean Object interaction policy agent completes subgoal task "Clean the mug in the sink." in unseen environment.
21

---

Under review as a conference paper at ICLR 2022
S
OPEN FRIDGE
PUT FRIDGE
CLOSE FRIDGE
OPEN FRIDGE
PICK APPLE
CLOSE FRIDGE
Figure 11: Cool Object interaction policy agent completes a subgoal task "Chill the apple in the refrigerator." in unseen environment.
TOGGLEON LAMP
Figure 12: Toggle Object interaction policy agent completes subgoal task "Turn the lamp." in an unscen environment.
22

---

Under review as a conference paper at ICLR 2022
HN
OPEN MIROWAVE
PUT MICROWAVE
TOGOLEON
Oor MICROWAYE
OPEN MICROWAVE
PICKUP BREAD
CLose MICROWAYE
(a) Heat Object
Figure 13: Heat Object interaction policy agent completes subgoal task "Heat the apple slice in the mi- crowave." in unseen environment.
SLICE BREAD
(a) Slice Object
Figure 14: Slice Object interaction policy agent completes subgoal task "Make slices of the bread." unseen environment.
23


Reviews:
[{"review_id": "ZQq8JfGEVoS", "reviewer": "Reviewer_KXWq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n- The work attacks a challenging problem\n- The idea is appealing and intuitive: making the  decomposing instructions into hierarchical subgoals and use different policy modules to handle them\n- Good empirical performance on competitive benchmark\n\n**Weaknesses**\n\n- Aside from the main claims, the paper lacks some critical details (see question 1) and is a bit confusing (see questions 2-4). See the questions below for concrete examples.\n- Hard to really pinpoint what the main contribution is. There are claims such as “novel language instructions” in sec 3.2.1 or “we propose a module” in sec 3.2.2. The above two examples are not mentioned in the main claims but they obscure the main points as I’m not sure if they are technical details or a contribution. Similar issue in the ablation study. The paper ablated on aspects that are not mentioned in the main claims such as DA, NIH, and OCMP. It’s unclear what conclusion the readers should draw from the analysis.\n- Lacks analysis compared with previous best models. The performance improvement seems large, but the paper provides little insight in terms of what gains it gets against similar hierarchical approaches (Zhang and Chai, 2021, Blukis et al., 2021). E.g., comparison in Table 2 is too weak (flat policy is a weak baseline); should compare to similar hierarchical approaches referenced above. \n- No significance tests\n\n**Questions**\n- Q1: There is prior work attempting to make instruction following hierarchical, which is duly cited in Related Work (e.g., Blukis et al., 2021). But I couldn’t parse the fundamental difference between “3 layer hierarchy” v.s. “1 layer hierarchy” in prior work. As far as I understand, prior work does generate subgoals which group low-level actions. If it’s pointing at the 3 layer modular structure, then I’d like to understand how having architectural hierarchy is useful in addition to action hierarchy \n- Q2: How is encoding the instructions with Bi-LSTM a novel approach? (sec 3.2.1)\n- Q3: Where does the subgoal space come from? Is this designed by the authors? \n- Q4: Why does <manipulate> appear again in the master policy? Shouldn’t this be for the PCC to decide?\n", "summary_of_the_paper": "The paper proposes a hierarchical approach (HACR) to tackle the long-horizon issue for embodied instruction following tasks. The hierarchical structure includes \n- a Policy Composition Controller (PCC): predicts a sequence of predefined subgoals\n- a Master Policy: decides whether to manipulate or keeps navigating\n- an Interactive Policy: performs the low-level interaction.\n\nThe main contribution claimed in the paper is the decomposition of the high-level goal into subgoals (which makes it interpretable), adding OEM and LEM to boost performance, and the SoTA performance on ALFRED benchmark.", "summary_of_the_review": "Overall, I think it’s a worthwhile paper if the empirical result is correct. The main weakness lies in the structure and the presentation of the paper which make it 1) hard to see the main contributions (is the main claim that hierarchical > flat or with the combination of the proposed components, MIP, OCMP, FIP, etc it works better than other hierarchical approaches?) 2) hard to draw insight as its comparison against prior best models is limited (e.g., does HACR also converge faster compared with other hierarchical approaches in Fig 4?). I would be happy to raise the scores if the authors could clarify more on the above two points as well as answering my questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QEr2FDcraug", "reviewer": "Reviewer_wh6P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- In one of their experiments, they show that training a hierarchical policy learns faster and more efficient action sequences as opposed to their flat policy counterparts. Also, hierarchical policy produces subgoals which leads to a more interpretable and transparent approach.\n-\tThe paper has good results with their method beating prior state-of-the-art methods on mostly all metrics. They also provide exhaustive ablations showing the importance of each component in their approach.\n- The paper is well-written and easy to follow with proper figures and with explanations of all the notations used in various modules. This is especially helpful since there are a bunch of moving parts in the approach. \n\nWeaknesses/Clarifications:\n\n- What was the value of $W$ chosen for experiments in eq (3)? I believe it is not mentioned in the paper. How was it decided and how does it impact the usability of Loop Escape module for the ALFRED task? \n-\tWhat are the 7 subgoals considered in the Policy Composition Controller module and how were they decided? It is unclear to me of how does the sub-instruction $\\hat{x}_i$ correspond to a single subgoal? For the sub-instruction (3) in language instruction in Figure (2) _“Turn around, bring the potato to the microwave on the right.”_, this corresponds to two subgoals “GoToLocation” and “HeatObject”.\n-\tIn figure (2), there are two “Manipulate” action blocks, one in PCC module and another as a predicted navigation action in Master policy. Based on the context, it seems that Manipulate in PCC passes the control to either Master Policy or Interaction Policies depending on whether the subgoal prediction from PCC was an interaction subgoal or not. But “Manipulate” in the Master Policy seem to be doing the same thing. How are they different from each other? It might be good to clarify. Also, it seems that “Manipulate” in Master Policy would act as a “Stop Token” to end the navigation action sequence, does there have to an interaction action after that? What would happen if PCC predicts 2 consecutive “GoToLocation” in a row, but the end of Master Policy triggers an Interaction Policy action due to the “Manipulate” action?\n", "summary_of_the_paper": "This paper proposes a hierarchical approach for policy learning in interactive instruction following for the ALFRED task. They infer a series of subgoals to be executed from the language instructions. Then, based on the subgoals predicted, they run Master policy which executes navigation actions, followed by Interaction policies which produce interactions to be done in the environment. They propose Object Encoding Module which provides target object information and serves as navigation subgoal monitor. They also propose a Loop Escape Module which based on the similarities between visual features detects if a loop has been made.", "summary_of_the_review": "Overall, I feel that this is a good paper with good results and clear presentation. I am happy to increase my score if my concerns can be answered.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FnXcZby0Pol", "reviewer": "Reviewer_4rTh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The hierarchical and modularized approach seems promising for addressing the ALFRED problem, which is a mixture of navigation and multiple interactive actions. Decoupling and specifying different reasoning processes, as well as using independent state representations, greatly reduces the state space in learning, and as claimed in the paper, improves generalization ability and interpretability. I believe the method could benefit and influence lots of future work.\n2. Comprehensive experiments in the main paper and in appendix nicely support the proposed methods and arguments in this paper. The performance of the agent on ALFRED has been improved by a significant margin.\n3. The overall paper is nicely written, tables, figures are clearly presented.\n\nWeaknesses / Comments:\n1. A common limitation of learning specialized network for each subgoal is the increase in hardware cost, training cost and sometimes computational cost, especially when people want to design more complicated interaction policy or when there are a large number of subgoals. I am wondering if the authors can discuss about this point (and perhaps provide some quantitative analysis on this issue).\n2. Following the previous comment, I believe the learning of some subtask can benefit each other, a simple example might be: heat and cool both requires the agent to put an object in to something, which such information is not shared across specialized policies. I would like to learn about the authors’ thoughts on this point.\n3. There are several points which are unclear to me: \n(1) Section 3.2.1, how to combine step-by-step instructions into a subtask instruction? I think ALFREAD only provide step-by-step instructions. I can understand this can be easily done in training because we know the ground-truth action correspond to each step-by-step instruction, but what about the validation data?\n(2) Following previous point, it is unclear how does each step-by-step instruction x^hat_i translate to multiple subgoals. E.g. Figure 2 Instruction 3 contains navigation action and object interaction, but from Equation 1, it seems that each x^hat_i only translates into a single action. Please correct me if I misunderstood anything.\n(3) The training of the seven interaction policies, e.g., training objectives. The paper also mentioned only two epochs of training is required, does it mean that the actions for completing each subgoal is similar (can be easily memorized? Or is it due to the limitation of the dataset?) I hope the authors can elaborate more on this point.\n", "summary_of_the_paper": "This paper proposes a hierarchical and modularized framework for addressing the ALFRED language-guided task completion problem. The method divides instructions into navigational and interactive subgoals, where the navigation and each interaction is processed by a specialized policy. To further improve the agent’s performance, the paper introduces object encoding module and loop escape module. The proposed methods are novel and have been experimented comprehensively in this paper, which could benefit a range of future research. ", "summary_of_the_review": "The paper is nicely written, it presents interesting and valuable ideas, and the experiments are comprehensive. I am happy to accept the paper if the authors can clarify my questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DlWJWY6Mqqk", "reviewer": "Reviewer_BjKc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Short version\nOverall, I think this paper presents an effective hierarchical approach to instruction following that’s tailored a bit too tightly to the ALFRED benchmark. I’m not too familiar with the VLN line of work, but I believe the authors have made a fair comparison to the existing works, based on the quantitative comparisons against baselines. Algorithmically, there seem to be a lot of moving parts that make the whole work well and achieve SOTA numbers, but can be interpreted as either clever, crafty or hacky, depending on how you look at it. I respect the effort put by the authors in carefully designing each component and engineering an effective algorithm for the ALFRED benchmark but have concerns regarding the generality of the findings.\n\n---\n## Strengths\n1. The writing and presentation is very effective and the authors do a great job of explaining the components of the algorithm in an intuitive manner.\n2. The ablations, although a bit binary, are exhaustive and do a great job of highlighting the importance of each component in isolation. The ablations are conducted like a well-written CVPR/ICCV paper, which tend to have some of the most thorough ablation experiments.\n3. The task that the paper addresses is extremely visual and the authors do a good job of visually showing rollouts and their method in action. It really highlights the long-horizon and semantic nature of the problem and intuitively describes the method (fig 1, 5).\n4. The implementation and experimental details, while not exhaustive, feel sufficient to understand and replicate the settings in which the results are presented. The impact of the paper will be greatly improved by a public code release.\n\n---\n## Weaknesses/Concerns\n\n1. My biggest concern with the method is that it seems overfit to the ALFRED benchmark and a lot of the design decisions were motivated by common failure cases in the benchmark. While that can be a great way to make progress on a challenging task like embodied instruction following, I’m not convinced that the algorithm and insights necessarily generalize to the broader range of problems. Specifically, the “loop escape” component or the “OEM module”, while solving real problems, feel a bit too specific and poorly justified. This is also depicted in how these components are motivated — e.g. “”To further improve the task completion performance, we propose…”.\n2. The authors make several claims that seem poorly motivated or not justified:  \n\t(i) [Section 3] “navigation and interaction must be distinguished because the former requires a broader set of information” — this could benefit by a proper discussion or references to prior work. While it’s true that almost all prior/concurrent work deals them independently, it’s almost a _bug_ and not a _feature_; we could use systems that better integrate these two capabilities. I am not saying that the choice of treating them independent is a wrong one (it does make things easy), but the claim needs to be qualified/justified a bit.  \n\t(ii) [Section 3.2] “this necessitates an independent module that can…” — why is this the case? This feels more like a by-product and not requirement.  \n\t(iii) [Section 4.3] “… highlights the modular structure’s benefits in long-horizon planning and superior generalization capabilities” — how does it do so? It’s not clear to me how the single policy metrics imply this. A bit more clarification would help here, since “better generalization” is presented as a key advantage of the proposed method at the start.  \n\n3. [Section 3.2.4] Regd. the loop escape module: $W$ being a hyperparameter feels a bit crafty. The job of the “loop escape” is to break out of a deadlock, and being able to do so for a fixed/preset value of repetition (which must be set for “best results”) again sounds like something that is overfit to the benchmark. In a general set of environments, this value could vary drastically and there could be a better way to handle this (if at all such a module is necessary).\n\n4. [Section 3.1] Regd. the policy composition controller: I get the impression that the benefits of this are overstated, and the main purpose of this controller is to learn correspondence between “semantic subgoals” and text instructions. According to Eqn. 1, these subgoals seem to be predefined, and hence, the mapping is pretty straightforward when done in a supervised manner and isolated for the controller. The claims about interpretability need to be toned down in this case.\n\n5. [Section 3.1] I had another question regarding the PCC: Based on the description, it seems to me that the PCC learns a one-to-one mapping from $\\hat{x}_i$ to subgoals $s_i$. Is that correct? If so, how does this handle instructions that need multiple subgoals, e.g. “heat the potato in the microwave” would need HeatObject and GoTo etc. Either way, a bit more clarification on how the instructions are handled could be useful in either the main text or appendix.\n\n6. While this may not be an outright weakness, a lot of the proposed contributions lead the work towards a “classical” pipeline, with individual stages for parsing inputs, perception/object segmentation, running control, loop detection/closure etc. Looking into the progress made in embodied AI in the robotics and vision communities, a lot of these _hand-engineered_ stages have benefitted from being replaced by data-driven means and going back to such a pipeline is intriguing —  it brings up the age-old question about what to learn v/s what to bake in as an inductive bias. While more instrumentation like the proposed method seems to improve benchmark metrics, it might come at the cost of “generalization of insights”. This relates to the earlier point (1), and a discussion about this philosophy would be nice. (I don’t hold this against the paper and this does not factor in my assessment, just a useful discussion to include.)\n\n\n---\n## Misc. Comments/Typos\nI noticed a couple of typos in the text that you can fix. Don’t worry, this does not affect my impression of the otherwise well-written paper :)\n\n- [Section 1] “To evaluate our proposal in a challenging scenario*s*”\n- [Figure 2, caption] “comprises of *tree* main components”\n\n\n---\n\n*Update*: Updating my recommendation to reflect the discussion by the others.", "summary_of_the_paper": "The paper looks at the problem of instruction following (using language) in a navigation+interaction task setting, encapsulated by the ALFRED benchmark. The authors propose a hierarchical modular approach that operates on three levels: (i) identifying subgoal sequence, (ii) a navigation “master” policy, and (iii) a manipulation/“interaction” policy. Evaluations against a slew of baselines from the vision+language navigation literature are presented, showing that the proposed method (HACR) outperforms existing baselines in both seen and unseen tasks. Careful ablations are also presented.", "summary_of_the_review": "The paper presents an effective algorithm for instruction following within the ALFRED benchmark but the means and insights seem overfit to the metrics. I am on the fence regarding acceptance and look forward to a response from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8_7qA6XHrWV", "reviewer": "Reviewer_pj1s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**2. Strengths**\n\nThe idea of hierarchical strategy sounds technically correct and the ablation study in the experimental section shows its better performance than the previous method. The method also provides more transparent execution processing since the strategy predicts the subgoal during inference besides the step-by-step instruction.\n\n**3. Concerns**\n\nThe reviewer has many concerns about the method which the reviewer thinks resolving them will make it a better paper.\n\n**(1). About the module design.**\n\nThe PCC and MP modules do not make any use of the visual input for subgoal prediction and navigation action prediction. This looks strange since it will always be easier for humans to see the room before making subgoal and action predictions. It will be better to verify visual input's effectiveness with experiments. At least, the reviewer thinks that a discussion of the usage of visual input for PCC and modules are needed.\n\n**(2). About the fairness of the comparison.** \n\nThe method uses additional supervision signals (the subgoal labels) compared with Singh et al (ICCV 2021). Also, it adopts a low-level text instruction as input when compared with HLSM. It's OK to use these additional signals as input but the reviewer thinks that it will be better to clarify them in the comparison.\n\n**(3). About performance comparison.**\n\nThe performance gap in the table between HLSM and HACR is small (16.70 to 16.29). We care more about the unseen testing split and so does the official Alfred challenge. The authors of HLSM recently release their code publicly on github (https://github.com/valtsblukis/hlsm.git) and achieve much better performance (20.27) with better parameters fine-tuning. It is OK not to compare with the updated performance in this submission since the code has not been released at that time. However, a discussion about how HACR is better and complementary to HLSM will be more convincing. Note that HLSM uses less supervision signals.\n\n**(4). About the overfitting issues.** \n\nThe performance of HACR seems to suffer from strong overfitting in table 1 (from seen to unseen and from validation to test). More analysis for such a performance gap is needed and makes the paper stronger.", "summary_of_the_paper": "**1. The main idea of the paper**\n\nThis paper introduces a hierarchical strategy for interactive instruction following. Following previous module networks[Singh et al., ICCV 2021 and Corona et al. 2020] for interactive instruction learning, the author adopts module networks for action prediction. Different from the previous module networks, the authors propose to learn the middle-level goals first and decompose the step-by-step action predictions into two modules, master policy for navigation and interaction policy for object interaction. To better improve the performance, the authors also provide some minor modules like the object encoding module and the loop escape module. The method overall achieves slightly better performance than the previous method.\n\n", "summary_of_the_review": "According to the concerns in the **Main Review**, the author currently tends to reject the paper. The viewer also believes addressing the issues will make the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper led to significant discussion, and the AC is generally on the fence. First of all, thanks to the reviewers for the significant time they invested in the discussion, and thanks for the authors for promptly and patiently answering our questions. 

Overall, the reviewer recommendations are positive. However, the discussion showed that despite the positive recommendation, the reviewers struggled to distill the general contribution of the paper beyond performance on ALFRED. In discussion, the authors distinguished their contribution from existing work by focusing on using a set of low-level policies at the root of the overall policy. This relies on the discrete set of behaviors that is defined within the ALFRED benchmark. It's not clear how it generalizes to the actual problem of instructing a robot to execute natural language instruction. In realistic scenarios, is it possible to define a set of behaviors in such a clean way, and at scale? And then train/manage a separate model for each behavior? The set of interaction policies in Figure 2 illustrates this challenge well. The answer to this scaling question is not clear. This corresponds to a concern raised repeatedly by the reviewers about the approach too specialized to ALFRED. The AC shares this concern. 

(which are roughly equal to the SOTA at the time of submission, but show significantly more overfitting to seen environments)

On the positive side, this is solid work, with good results. The paper is well written, and the authors largely addressed the concerns raised as much as possible. The results are not SOTA though. The current SOTA was submitted on 09/19/2021, prior to the ICLR deadline -- it's not included in the results table in this paper. (To clarify, the fact that it's not the current SOTA does not affect the final decision, as they are considered as contemporaneous.) With concerns regarding the specificity of the approach, this paper may interest researchers working on ALFRED, but not clear to what depth, despite the clearly significant work and effort the authors put into the paper. 

(If the paper is accepted, the AC asks the authors to fix the standing errors with regard to previous work, as discussed below, and to include more recent results from the leaderboard)
Decision: Reject