Egocentric Planning for Scalable Embodied Task Achievement
Authors: Xiaotian Liu
Hector Palacios
Christian Muise
Venue: NeurIPS 2023 poster (2023)
Keywords: Embodied AI
High-Level Actions
Symbolic Reasoning
Replanning
ALFRED Challenge
Flexible Task Achievement
User-Goal Understanding
Object Types and Actions
Perception Grounding
TL;DR: In high-level environments, flexible embodied AI agents leverage known object types and actions for grounding perception and understanding user-goals. Our fast, iterative symbolic replanning won the ALFRED challenge

Abstract: Embodied agents face significant challenges when tasked with performing actions in diverse environments, particularly in generalizing across object types and executing suitable actions to accomplish tasks. Furthermore, agents should exhibit robustness, minimizing the execution of illegal actions. In this work, we present Egocentric Planning, an innovative approach that combines symbolic planning and Object-oriented POMDPs to solve tasks in complex environments, harnessing existing models for visual perception and natural language processing. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability, achieving an impressive 36.07\% unseen success rate in the ALFRED benchmark and winning the ALFRED challenge at CVPR Embodied AI workshop. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions, as well as what object types reveal information about others. It can naturally scale to solve new tasks beyond ALFRED, as long as they can be solved using the available skills. This work offers a solid baseline for studying end-to-end and hybrid methods that aim to generalize to new tasks, including recent approaches relying on LLMs, but often struggle to scale to long sequences of actions or produce robust plans for novel tasks.

Egocentric Planning for Scalable Embodied Task Achievement
Xiaotian Liu" ServiceNow Research Montreal, QC. Canada xiaotian. 1iu @mail, wtw.onto..ccca
Hector Palacios ServiceNow Research Montreal, QC, Canada hectorpal @gmail com
Christian Muise Queen's 's University Kingston, ON, Canada christian www.si..iiiiieeeee....m @queensu.co
Abstract
Embodied agents face significant challenges when tasked with performing actions in diverse environments, particularly generalizing across object types and ex- ecuting suitable actions to accomplish tasks. Furthermore, agents should exhibit robustness, minimizing the execution of illegal actions. this work, we present Egocentric Planning, innovative approach that combines symbolic plamning and Object-oriented POMDPs solve tasks in complex environments, harnessing ex- isting models for visual perception and natural language processing. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability, achieving impressive 36.07% unseen success rate in the ALFRED benchmark and winning the ALFRED challenge at CVPR Embodied AI workshop. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions, well what object types reveal information about others. It can naturally scale to solve new tasks beyond ALFRED, as long as they can be solved using the available skills. This work offers a solid bascline for studying end-to-end and hybrid methods that aim to generalize to new tasks, including recent approaches relying on LLMs, but often struggle scale to long sequences of actions or produce robust plans for novel tasks.
Introduction
Embodied task accomplishment requires an agent process multi-modal information and plan over long task horizons. Recent advancements in deep learning (DL. models have made grounding visual and natural language information faster and more reliable (MBP+21). As a result, embodied task-oriented agents have been the subject of growing interest 2TG+ YRT+22 WDKM21). Benchmarks such as the Action Learning From Realistic Environments and Directives (ALFRED) were proposed to test embodied agents' ability to act in unknown environment and follow language instructions or task descriptions (STG+20) The success of DL has led researchers to attempt end-to- end neural methods (ZC21) SGT+21 In environment like ALFRED, these methods mostly framed as imitation learning, where neural networks are trained via expert trajectories. However, end-to-end optimization leads to entangled latent state representation where compositional and long-horizon tasks are challenging to solve. Other approaches use neural networks to ground visual information into persistent memory structures to approaches rely on templates of existing tasks, making them difficult to generalize to new problems unexpected action outcomes. ALFRED agents must navigate long horizons and skill assembly, operating within a deterministic environment that only changes due to the agent's 's actions.
21). information (MCR+2) BPF+21 (1)++1). These
"Work done when Xiaotian Liu intern ServiceNow Research. Work done when Hector Palacios was a rescarch scientist at ServiceNow Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Long sequential decision problems with sparse rewards are notoriously difficult to train for gradient- based reinforcement learning (RL) agents. But a symbolic planner with a well-defined domain can produce action sequences composed of hundreds of actions in less than a fraction of a second for many tasks. Embodied agents might need to conduct high-level reasoning in domains with long action sequences. In such scenarios, human experts can opt to model the task within lifted planning language. Planning languages such as Planning Domain Definition Language, or PDDL (HLMM19, is the most natural candidate, given the high scalability of planners that use such languages (GB13) For instance, we define the tasks of a home cooking robot with actions, such as opening, move-to, pick-up, put-on, and objects, such as desk, stove, egg found in common recipes.
However, most symbolic planning techniques are relatively slow in partially observable environments with a rich set of objects and actions. Thus we propose an online iterative approach that allows our agent to switch between exploration and plan execution. We explicitly model the process of actively gathering missing information by defining a set of unknown-objectts and exploration-actioos that reveals relevant information about an agent's 's goal. Instead of planning for possible contingencies, we use an off-the-shee classical planner(HofOD] for fully-observable environments to determine whether the current state has enough knowledge to achieve the goal. If more information is needed, our agent will switch to goal-oriented exploration to gather missing information. Our approach can seamlessly incorporate any exploration heuristics, whether they are learned or specified.
We implemented and evaluated our method on the popular embodied benchmark ALFRED using only natural language task descriptions. To demonstrate the effectiveness of our method, we use the same neural networks for both visual and language grounding used in FILM, which was the SOTA and winner of the 2021 ALFRED challenge (MCR+22). See a comparison with other methods in Table D By replacing FILM's template-basee policies with our egocentric iterative planner, our method improved FILM's success rate (SR) by 8.3% (or a 30.0% relative performance increase) in unseen environments winning the the ALFRED challenge at CVPR 2022 Embodied AI workshop. Our empirical results show that the performance increase was attributed to a more robust set of policies that account for goal-oriented exploration and action failures. We also show that our method can conduct zero-shot generalization for new tasks using objects and actions defined in the ALFRED setting.
2 ALFRED Challenge
ALFRED is a recognized benchmark for Embodied Instruction Following (EIF), focused on training agents to complete household tasks using natural language descriptions and first-person visual input. Using predetermined set of actions, the agent operates in a virtual simulator, A12Thor (KMG+17), to complete a user's task from one of seven specific classes instantiated in concrete objects. For example, a task description may be "Put a clean sponge on the table," " and instructions that suggest steps to fulfill the task. The agent must conduct multiple steps, including navigation, interaction with objects, and manipulation of the environment. Agents must complete a task within 1000 steps with fewer than 10 action failures. We base our approach on task descriptions rather than the provided instructions. Task complexity varies, and an episode deemed successful all sub-tasks are accomplished within these constraints. Evaluation metrics are outlined in Section 8.
3 Overview of Egocentric Planning for ALFRED
Figure [ Iillustrates our proposed method integrating several components to facilitate navigation and task completion: a visual module responsible for semantic segmentation and depth estimation, a language module for goal extraction, a semantic spatial graph for scene memorization, and an egocentric planner for planning and inference. Initially, high-level language task description is utilized to extract goal information, and the agent is provided with a random exploration budget of 500 steps to explore its surroundings. The random exploration step is used to generate diverse set of object cluster for further exploration using our planner. Subsequently, at = 500, the gathered information from the semantic spatial graph is converted into PDDL problem for the agent. We employ a novel open-loop replanning approach, powered by an off-the-she planner, to support further exploration and goal planning effectively.

---

Directed Graph
NL Instraction
Nodes:
Put
K3.0:
(d3) Mack Depih, Chacs,
Language Module
Edges:
Agznin Left Rightt
Segmentation
Goal: (On Clock Table)
Iritial Condition: (Oi Bread lable)
(Tomato (sgent 3.51)
Depth Vision Medule
Spatial Semantic Graph
Egocantric
Sensory Input
a.esp,
Environment
Plan Action plan,
Figure Egocentric Planning for ALFRED
To facilitate egocentric planning, we first define the ontology of the planning problems, which encompasses action schemas, object types, and potential facts schemas. The natural language task description is subsequently converted into planning goal. Following the initial exploration phase, semantic spatial graph updated to indicate the new position of the agent and its perceptions. The algorithm iterates through these steps: a) finding plan for achieving the goal and returning upon success; b) if no plan exists, replacing the goal with another fact, called (explore), associated with unvisited states. The semantic spatial graph updated during each iteration to construct the new initial state of the agent, allowing for an incremental egocentric view of the environment.
4 Embodied Agents: Generalization within Object Types and Skills
Embodied Agents are task solvers environments linked to hidden Object-oriented POMDP, emphasizing objects, actions/skills, and costs rather than rewards. We assume that these objects mediate perceptions and actions. An Environment is a tuple, & (Ae, Te, Ve, reset, step), with Ag being the set of parameterized actions, Te defining the of object types and potential values Ve. reset representing the first observation, and step representing an action execution that returns an observation and cost, A Task represented by tuple, TE (Lg, Gg). which sets the initial state and the goal of the environment via the reset function. Lastly, Embodied Agents employ functions M, to initialize their mental state and Mr to update it, have a policy иM to select actions and a function GM for deciding when they consider the goal to be achieved.
Agents that tackle stream of tasks within the known object types and their inherent capabilities expressed through parameterized actions. Such over-specializattin provides opportunity for developing robust agents that can generalize across different tasks within the same environment. This principle applies to physical and software agents whose actions are grounded in these objects. Our framework forms basis for creating flexible and adaptable agents that can perceive and act upon recognized object types. Further details and implications are covered in Section
environment have incentives to specialize the
Egocentric Planning for Embodied Agents
Here, we introduce Egocentric Planning, an approach for embodied agents to solve tasks in Object- oriented POMDP environments. The approach requires symbolic planning model that assumes full observability for testing both action applicability and goal achievement. However, the complexity of specifying or learning such symbolic model is simplified by its alignment with the Object-oriented POMDP environment, object types, and action signatures. Egocentric planning relies on reasonable assumption to derive sensor model, equivalent to function that updates the distribution of possible

---

current states given new observations (AGG09). For instance, in planning domain with objects for different types and colors, the sensor model can say that cubes and rectangles are red, while ovals and circles are black. When the agent examines an object, and observes that it is black, the agent should filter the belief to represent that the object cannot be a cube or a rectangle. In contrast, our method requires set of anchor object types that are assumed to reveal information about other objects and exploration actions that allow visiting new anchor objects, revealing new objects and their properties. The method can be combined with standalone models for processing images or natural language. The resulting method is both theoretically sound, assuming that the symbolic planning model is correct, and practically applicable, it leverages the speed of symbolie planners.
We present abstract algorithm that lever- ages the objects types and action signatures of the Object-oriented POMDP to ground the required symbolic planning model.
Algorithm 1
Iterative Exploration Replanning (IER)
Input: Environment (Ag. Te. Ve, reset, step)
Input: Planning domain PDD (T, V, A., $)
5.1 Background: Parameterized Full-Observable Symbolic Planning
Input: Anchor Types & Exploration Acts (Ta..t)
Input: Mental state Init & Update (MPD) My
In the supplementary material, Section E. we define Parameterized Full- hall-Observaabe Sym- bolic Planning domains and problems. A planning domain defined as a tuple PDD
Input: Task (Je, Gg)
Output: Successful trace T or Failure
I: p t reset(Le, [egGg
Initial perception
MPP 2: O,I,G (MG++MBBB (p. L3, Gg)
(T, V, P.A. A, 2) where T is finite set of ob- ject types with V a finite set of values, P is finite set of typed predicates, A is finite set of typed actions with preconditions and effects, and & is function that maps each action and predicates to a tuple expressing their typed arguments. planning problem consists of a planning domain, a finite set of objects, and descriptions of the initial state and the goal.
3: C {o' [type(d') Ta and o' occurs in /} D C: Observed Anchor Objects
4: 7. s. x- Falsc
5: while not solved do
6:
Tsoine +- -Sollee(PD,, O,I,G))
7:
if Toouue is None then
8:
Ae A for o E O and type(o) E To do o of then
9:
11:
+ IU {(unknown o)} if (unknown o) E I then A + A. U (ExploreAct(a, o)
For ALFRED, we rely Classical Planning, a kind of full-observable planning that as- sumes deterministic actions, where precondi- tions are a set of predicates, and effects are a pair of sets of predicates called add and delete, representing the action's 's modifications. Cor- respondingly, a parametric classical planning problem is full-observable planning prob- lem where the initial state and the goal are conjunctions of grounded predicates.
12:
13:
for X)
14:
Ge + {(explored) Texplore
15:
Solve( (PD with Ae,
16:
return Failure if Templore is None
17:
for a E Tezplore do
18:
p, (
step(a)
19:
Break if failed E T.append(a)
20:
5.2 Egocentric Planning for Embodied Agents
21:
+ M (O,I,a,p,c)
22:
if E X then
Egocentric planning employs a user-specified fully-observable symbolic planning model to address tasks in Object-oriented POMDP en- vironments. In the next section, we detail the specifics of our ALFRED implementa- tion. As actions in ALFRED are determinis-
23:
C C+ (tpe(otpp)oo)))) E To and o' argument of a)
24:
else
25:
for E solve do
26:
p, C t (a) Break if failed E p T.append(a)
27:
tic, we can apply a classical planner to solve
28:
the associated planning problems. We first
29:
O, M a,,,,,
present a general algorithm for parametric full-observable planning problems.
30:
solved + True I satisfies G
31: return
Algorithm immllemntts our Egocentric Planning approach by constructing sequence of planning problems, alternating exploration and task solving. each step, the agent's mental state updated though sensory input to represent the current state and the known objects that will be part of the

---

following planning problem. The agent deems the goal achieved when it finds a plan for achieving it and executes the plan successfully. Our policy (TT) leverages a standalone planner for both exploration and task solving.
Our exploration and sensing strategy hinges on two elements. First, anchor object types Ta E T are types of objects that reveal the presence of others. For example, in ALFRED, location is the sole anchor object type. Second, exploration actions X' S A enables the discovery of previously unknown objects. For instance, moving to a new location in ALFRED enables observing new items and locations. This object-centric view of the environment simplifies the creation of a symbolic model required for planning. The function ExploreAct (a. o) returns copy of the function where (unknown o) is added to the precondition, and the effect is extended to cause (explored) to be true and (unknown o)) to be false.
Our agent's mental state is represented by set of objects and initial state of grounded predicates, (O, J). The function MPP initializes the symbolic goal G and sets (O, 7) with the initial objects and true grounded predicates. The mental state update function, MPP. updates (O, I) given the executed action a, the returned perception p. and the cost e of executing a. For example, the mental state update in ALFRED adds new objects observed in the perception. It updates the initial state with these new objects and their properties, including their relationship with non-perceived objects like location. Both functions maintain the alignment between the current planning state and the environment with an underlying object-oriented POMDP model.
Specifically, the actions, object types, and the values of the environment may be different from the planning domain's. While the ALFRED environment supports movement actions like move-forward or turn-left, t, the planning domain might abstract these into a move action. Although the ALFRED environment does not include location type, we manually specify it movable objects can be associated with a specific location. Indeed, the first initial state I includes a first location and its surrounding ones. Thus, the agent can revisit the location once it has found solution involving that object. While integrating these components may be challenging, it is a hurdle common to all embodied agent approaches aiming to generalize across components under constrained training budget. In general, practical applications will likely monitor the agent's 's performance regarding object types and skills. We discuss the advantages and drawbacks of our approach in Section 9
6 Approach for ALFRED
6.1 Model Overview
Our method is structurally similar to FILM but we substitutes the SLAM-style map with graph structure for object and location information storage (MCR+22) The action controller employs our iterative egocentric planner that utilizes semantic location graph for task completion and exploration. The node of the graph are uniquely labeled by the coordinates (x-coordinate, y-coordinate, facing direction), and the edges are labeled by actions(turn-rihtt, turn-left, move-forward). Each node in the graph stores object classes, segmentation masks, and depth information generated by the vision module. We label the nodes with "known" if the agent has visited the particular location and "unknown" " if the node is only observed. Unknown nodes on the graph are prioritized for exploration based on objects they contain. This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks. etch timestep, the vision module processes an egocentric image of the environment into depth map using U-Net, and object masks using Mask2-RCNN (HDDG17 LCQ 18). The module then computes the average depth of each object and stores only those with an average reachable distance less than 1.5. A confidence score, calculated using the sum of the object masks, assists the egocentric planner in prioritizing object interaction. We employ text classifier transform high-level language task description into goal conditions. Two separated models determine the task type and the objects and their properties. For instance, task description like "put an apple on the kitchen table" would result in the identification of "pick-up-ande and-place" task and a goal condition of (on apple table). We convert the PILM-provided template- based result into goal conditions suitable for our planner. The spatial graph, acting as the agent's memory during exploration, bridges grounded objects and the state representation required for our planner. The graph, updated and expanded by policies produced by our egocentric planner, encodes location as the node key and visual observations values, with edges representing agent actions.
5

---

6.2 Egocentric Agent for ALFRED
The coject-oriented environment for ALFRED, denoted as 8 (Ag. Te, Vg, reset, step), includes actions such as pick-up. put-down, toggle, slice, fove-forward, turn-left, and tumnrightt. The FILM vision module is trained as flat recognizer of all possible object types. We postprocess such detection into high level types Object and Receptacle in Te, with corresponding Ve indicating subtypes and additional properties. For instance, the object type includes subtypes like apple and bread, while Receptacle includes subtypes such as counter-top and microwave. Values incorporate properties like canHeat, canSlice, isHeated, and isCooled.
The planning domain for ALFRED is denoted PD = VT, T includes the Location type, with corresponding values in V representing inferred locations as actions deterministic. Predicates in P represent the connection between locations and the presence of objects at those locations. Further predicates express object properties, such as canHeat, canSlice, isHeated, and isCooled. Unlike the ALFRED environment, actions in the planning domain are parametric and operate on objects and locations. Instead of move-forward. the planning domain features a move action, representing an appropriate combination of fore-for turn-left, and turn-right. For actions like toggle in ALFRED, which have different effects depending on the object in focus, we introduce high-level, parametric actions like heat that can be instantiated with the relevant object. As the actions in ALFRED are deterministic, we use a classical planner, enabling high scalability by solving one or two fresh planning problem per iteration of Algorithm |
7 Related Work
Visual Language Navigation (VLN) involves navigating in unknown environments using language input and visual feedback. In this context, we focus on the ALFRED dataset and methods based on the Al2-Thor simulator, which serve as the foundation for various approaches to embodied agent tasks. Initial attempts on ALFRED employed end-to-end methods, such as Seq2Seq model (STG+20), and transformers (ZC21). However, due the long episodic sequences and limited training data, most rop-performing VLN models for ALFRED employ modular setups, incorporating vision and language modules coupled with a higher-level decision-making module (MCR 22, MC22] (O22). These methods typically rely on hand-crafted scripts with a predefined policy. However, this approach has several downsides. First, domain experts must manually specify policies for each new task type, regardless of the changes involved. Second, fixed-policy approaches do not allow the agent to recover from errors in the vision and language modules, as the predefined policy remains unchanged.
Recently, there has been growing interest in utilizing planning as a high-level reasoning method for embodied agents. Notable work in this area includes OGAMUS + 22), and DANLI (2))+22).
These methods demonstrate the effectiveness of planning in achieving more transparent decision- making compared end-to-end methods while requiring less domain engineering than handcrafted approaches. However, these works primarily tested in simpler tasks or rely on handcrafted planners specifically tailored to address a subset of embodied agent problems.
Large Language Models (LLMs) have recently been harnessed to guide embodied actions, offering new problem-solving approaches. Works like Ichter al.'s SayCan utilize LLMs' likelihoods and robots' affordances, grounding instructions within physical limitations and environmental context (iBC+23). Song et al.'s (SWW- + +23) LLMM-Planeer and Yao et al.'s (YZY+23) ReAct have further demonstrated the potential of LLMs for planning and reasoning for embodied agents. Alternative methods using LLMs include Jia et al. and Bhambri et al.'s hierarchical methods utilizing task instructions (ttritonn[[[tt222 22 (B2M)22). However, the scalability and robustness of these methods is under study. Our approach, in contrast, ready capitalize the expressivity of objects and actions for superior generalization in new tasks, ideal for rich environments with multiple domain constraints. This challenges of using LLM for symbolic planning noted by Valmeekam al. (VSM 23) who showed the weakness of prompting-basee approaches, and Pallagani et al. (PMM+22 who highlight the massive amount of data required for reaching high accuracy.
8 Experiments and Results
The ALFRED dataset contains a validation dataset which split into 820 Validation Seen episodes and 821 Validation Unseen episodes. The difference between Seen and Unseen is whether the room

---

Test Seen
Test Unseen
SR
GC
PLWSR
PLWGC
SR
GC
PLWSR
PLWGC
Seq2Seq
3,98
9.42
2.02
6.27
0.39
7.03
0.08
4.26
ET
38.42
45.44
27.78
34.93
8.57
18.56
4.1
11,46
HLSM
25.11
35.15
10.39
14.17
24.46
34.75
9.67
13.13
FILM
28.83
39.55
11.27
15.59
27.8
38.52
11.32
15.13
LGS-RPA
40.05
48,66
21.28
28.97
35.41
45.24
15.68
22.76
EPA
39.96
44.14
2.56
3.47
36.07
39.54
2.92
3.91
Prompter
53.23
64.43
25.81
30.72
45.72
58.76
20.76
26.22
Table 1: Comparison of our method with other methods on the ALFRED challenge. The challenge
declare winner the method with higher Unseen Success Rate (SR). For all metrics, higher is better. EPA is our approach. Under the line are approaches submitted after the challenge leader board closed.
environment available in the training set. We use the validation set for the purpose of fine-tume fine-tuning. The test dataset contains 1533 Test Seen episodes and 1529 Test Unseen episodes. The labels for the test dataset are contained in an online server and are hidden from the users. Four different metrics used when evaluating ALFRED results. Success Rate (SR) is used determine whether the goal is achieved for a particular task. Goal-condition Success is used to evaluate the percentage of subgoal-condittons met during episode. For example, if an objective is "heat apple and put it on the counter," then the list of subgoals will include "apple is heated" and "apple on counter." The other two metrics are Path Length Weighted by Success Rate (PLWSR) and Path Length Weighted by Goal Completion (PLWGC), which are SR and GC divided by the length of the episode. For all the metrics, higher is better. We compare our methods, Egocentric Planning Agent (EPA), with other top performers on the current AL ALFRED leaderboard. Seq2Seq and ET are neural methods that uses end-to-un aad-to-end training (STG 20 ZC2I). HLSM. FILM. LGS PRRA are hybrid approach that disentangles grounding and modeling 2BPF'21
MCR 22
MC22 (022).
8.1 Results
[ comparison of our outcomes with other top performing techniques on the ALFED dataset. With an unseen test set success rate of 36.07% and a seen test set rate of 44.14%, our EPA method won the last edition of the ALFRED challenge. The ALFRED leaderboard ranks based on SR on unseen data, reflecting the agent's capacity to adapt to unexplored environments. At the moment of writing, our achievements rank us second on the ALFRED leaderboard. The only method surpassing ours is Prompter(1022), which utilizes the same structure as FILM and incorporates search heuristics- findings indicate that the performance enhancements are largely attributed to increasing obstacle size and reducing the reachable distance represented on the 2D map. The other top performer is LGS-PRA, which is also based the FILM architecture, that improves performance via techniques on landmark detection method and local pose adjustment. In contrast, EPA employs a semantic graph, a representation distinct from the top-down 2D map applied by FILM, HLSM, LGS-PRA and Prompter. Our findings suggest that the performance boost over FILM and similar methods stems from our iterative planning approach, which facilitates recovery from failure scenarios via flexible subgoal ordering. Techniques outlined in LGS-PRA and Prompter should be compatible with our approach as well. The integration of these techniques with EPA an interesting area to investigate for future works. Our method exhibits lower PLWSR and PLWGC due to our initial 500 exploration steps. These steps are convenient for our planner to amass diverse of of objects clusters and action angles to determine proper expandable initial state. Although pure planning- -baed exploration would eventually provide us with all the information required solve the current task, tends to excessively exploit known object clusters before exploring unknown areas. We only save object information when they are within immediate reach which means the agent acting on immediate observations. This is done for convenience of converting observation into symbolic state and to reduce the plan length generated by the planner. Additional integration with a top-down SLAM map should help decrease the initial exploration steps, consequently enhancing both PWSR and PLWGC.
prompting queries on an extensive language model. However, their own
7

---

Failure Modes
Seen
Unseen
Ablation
Valid Unseen
Valid Seen
Obj not found
19.36
23.54
SR
GC
SR
GC
Collison
9.14
11.34
Base Method
40.11
44.14
45.78
51.03
Interactions fail
7.33
8.98
w/ gt segment
45.13
49.72
49.22
53.21
Obj in closed receptele
18.21
16.33
w/ gt depth
41.85
46.03
48.98
51.46
Language error
17.92
21.31
w/ gt language
52.29
56.39
60.55
64.76
Others
28.04
18.5
w/ all gt
58.33
63.71
68.49
73.35
Table 2: Percentage Error of each types in the validation set.
Table 3: Ablation study with ground truth(w/ gt) of different perceptions
Ablation parameters
Valid Unseen
SR
GC
PLWSR
PLWGC
Base Method
40.11
44.14
2.04
3.71
w/o init observation
30.75
36.16
17.34
21.93
200 init observation
37.3
42.23
12.41
16.34
w/o object pref
32.89
35.95
3.05
3.81
w/o fault recovery
27.31
26.55
2.41
3.36
w/o symbolic planning
3.47
4.61
2.16
2.98
Table 4: Ablation study with different parameters
8.2 Ground Truth Ablation
Our model, trained on the ALFRED dataset, incorporates neural perception modules to understand potential bottlenecks in visual perception and language, explored via ablation studies detailed in 3 We observed modest enhancements when providing the model with ground truth depth and segmentation, with ground truth language offering the most significant improvement (18.22% and 19.87%). Our egucentric planner, responsible for generating dynamic policies through observation and interaction, proves more resilient interaction errors compared to policy-template based methods like HLSM and FILM (MCR + 22 BPF+21). Thus, having the correct interpretation of the task description impacts
significantly than ground truth perception. We examined common reasons for failed episodes using both the seen and unseen datasets. The most prevalent error (23.54% unseen) the inability of the agent to locate objects of interest due failed segmentation an unreachable angle. Errors stemming from language processing, where the predicted task deviates from the ground truth, are given priority. The results can be seen Table 2
method
8.3 Parameter Ablation
Neural vision modules may struggle with out-of-distributio data, such as ALFRED's unseen envi- ronments, leading to potential errors object segmentation and distance estimation. Unlike hybrid methods like FILM and HLSM, our goal-oriented egocentric planner reassesses the environment after failure, generating new action sequences. For instance, in a task to hold a book under lamp, a failure due to obscured vision be addressed by altering the action sequence, illustrated by the drop in performance to 27.31% % in our ablation study (Table
FIL.''s semantic search policy, trained to predict object locations using top-down SLAM maps, differs from our symbolic semantic graph representation, which precludes us from adopting FILM's approach. Instead, we hand-crafted two strategies to search areas around objects and receptacles based on common sense knowledge, which improved our Success Rate by 7.22% with no training. However, our agent only forms node when it visits a location, necessitating phase. Neglecting this step resulted in 9.35% performance drop. FILM's SLAM-based approach can observe distant objects, requiring fewer initial observation steps and leading better PLWSR and PLWGC. Combining a tow-down SLAM map with our semantic graph could mitigate the need for our initial observation step and improve these scores.
initial observation
Our algorithm's frequently re-plans according to new observation which is also essential for real- world applications using online planning system. Using modern domain- innuiinnendeeedeent planners, such

---

as Fast Downward and Fast Forward (HofOl Hel06), we've ve significantly reduced planning time. We tested a blind breadth-first search algorithm but found impractical due to the time taken. A 10-minute cut-off was introduced for each task, which didn't impact our results due to an average execution time of under minutes. Yet, this constraint severely lowered the success rate of the blind search method, only completing 3.47% of the validation unseen tasks 4).
8.4 Generalization to Other Tasks
Our approach allows zero-shot generalization to new tasks within the same set of objects and relationships, offering adaptability to new instructions without manually specify new task types and templates. While neural network transfer learning remains challenging in embodied agent tasks, and 'ILM's template system requires hand-crafted policies for new tasks, our egocentric planning breaks a task into a set of goal conditions, autonomously generating an action sequence. This allows our agent to execute tasks without transfer learning new task templates. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made minimize perception errors. We tested this five new task types not in the training data, achieving an 82% success rate, as shown in Table sectior(..2 of the Supplementary Material. We could also adapt to new objects, actions, and constraints, such as time and resource restrictions. However, the current ALFRED simulator limits the introduction of new objects and actions. This flexibility accommodate new goals is a key advantage of our planning- based approach.
Discussion
Our contribution poses Embodied Agents as acting in a Partially Observable Markov Decision Process (POMDP), which allows us tackle broad class of problems, and further expands on the potential applications of planning-based methods for embodied agents. By adopting this approach, our method offers more flexible and adaptable solution address the challenges and limitations associated with fixed-policy and handcrafted methods.
We found that using intentional models with factored representation in embodied agents is crucial. Whereas a learned world model provides a degree of useful information, their associated policies might under-perform for long horizon high-level decision-making unless they are trained on massive amounts of interactions (SSS+17), 17). Comparing a world model with classical planning uncovers interesting distinctions. While creating PDDL-based solution requires an initial investment of time, that time is amortized in contexts that require quality control per type and skill. On the other hand, the cost spent on crowdsourcing or interacting with a physical environment can be significant (STG+20)
Learning classical planning models is relatively straightforward with given labels for objects and actions (AFP* 18 CDVA 22). On the other hand, we aim to learn a world model, we should consider the performance planning time and end-to-end solutions may not be the ideal approach in this context. When compared to intentional symbolic-basee methods, end-to-end strategies tend to compound errors in more obscure manner. Furthermore, as learned world models rely blind search or MCTS, in new tasks the model might waste search time in a subset of actions that are irrelevant if action space is large, while symbolic planning can obtain long plans in few seconds. We also propose addressing challenging planning tasks using simpler planning techniques, providing foundation for more complex planning paradigms. Our method could be further enhanced by encoding explicit constraints and resources, such as energy.
While other methods might outperform our model in the ALFRED benchmark, they often limited in scope. For instance, despite its high unseen success rate, the Prompter model serves as a baseline rather than comprehensive solution for embodied agents. Our contribution provides method that can adapt to shifts in task distribution without changing the symbolic planning model, while also supports changes in the agent's 's capabilities by modifying the symbolic planning model.
We acknowledge certain limitations in our approach. While we support safe exploration assuming reversible actions, real scenarios beyond ALFRED could include irreversible actions and failures. Our model also sensitive to perception errors, making end-to-end methods potentially more robust in specific scenarios. Furthermore, perception model lacks memory, potentially hindering nuanced

---

belief tracking. Despite these challenges, our method offers promising direction for future research embodied agent applications. In section B
we expand on the discussion our approach.
10 Conclusion and Future Work
Our work introduces novel iterative replanning approach that excels in embodied task solving, setting a new baseline for ALFRED. Using off-the-shelf automated planners, we improve task decomposition and fault recovery, offering better adaptability and performance than fixed instruction and end-to-end methods. Moving forward, we aim to refine our location mapping for precise environment understanding, extract more relational data from instructions, and integrate LLMs as exploration heuristics. We plan to tackle NLP ambiguity in multi-goal scenarios by prioritizing and pruning goals based on new information. Moreover, we aim to upgrade our egocentric planner to manage non-deterninistic effects, broadening our method's scope beyond ALFRED. This direction, leveraging efficient planning methods for non-deterministic actions, is promising for future research (MBM14).
Acknowledgments
We want to acknowledge the ServiceNow research for providing the research opportunity and computational infrastructure conduct this research project. This work was also made possible due to the assistance of Cyril Ibrahim at ServiceNow during the experimental execution phases of our project. We would also like thank members of Mulab Queen's 's University for providing supporting research environment. Lastly, we want to acknowledge the assistance from organizers of the CVPR Embodied Agent Workshop for their guidance and feedback throughout the competition(DB) 22)..
References
[AFP+18] Ankuj Arora, Humbert Fiorino, Damien Pellier, Mare Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33(c20):1-25, 2018.
[AKFM22] Masataro Asai, Hiroshi Kajino, Alex Fukunaga, and Christian Muise. Classical planning deep latent space. Artif. Intell. Res., 74:1599-1686, 2022.
[ALPK18] Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling, Modular meta-learning, In Conference on robot learning, pages 856-868, PMLR. 2018.
[APG09] Alexandre Albore, Héctor Palacios, and Hector Geffner. A translation-based approach contingent planning. In Twenty-First International Joint Conference on Artificial Intelligence, 2009.
[BKMC22] Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Hierar- chical modular framework for long horizon instruction following, 2022.
[BPF+21] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRL,, volume 164 of Proceedings Machine Learning Research, pages 706-717. PMLR, 2021.
[CC04] Stephen Cresswell and Alexandra Coddington. Compilation of Itl goal formulas into pddl. In ECAI, volume 16, page 985, 2004.
[CDVA +22] Ethan Callanan, Rebecca De Venezia. Victoria Armstrong. Alison Paredes, Tatha- gata Chakraborti, and Christian Muise. Macq: holistic view of model acquisition techniques. In ICAPS 2022 Workshop Knowledge Engineering for Planning and Scheduling, 2022.
[CMW13) Stephen N. Cresswell, Thomas L. McCluskey, and Margaret M. West. Acquiring planning domain models using LOCM. The Knowledge Engineering Review, 28(2):195- 213, 2013.
10

---

[2] Rohan Chitnis, Tom Silver, Joshua B Tenenbaum, Tomas Lozano- Perez, and Leslie Pack Kaclbling. Learning neuro-symbolie relational transition models for bilevel planning. In 2022 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4166-4173, IEEE, 2022.
[DBB+22] Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X. Chang. De- vendra Singh Chaplot, Changan Chen, Claudia Pérez D' D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, and Jiajun Wu. Retrospectives on the embodied ai workshop, 2022.
[DCLT19]
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- training of deep bidirectional transformers for language understanding, In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171 1176.. Association for Computational Linguistics, 2019.
Jacob Devlin, Ming-
[DNR* 220 Srijita Das, Sriraam Natarajan, Kaushik Roy, Ronald Parr, and Kristian Kersting. Fitted q-learning for relational domains. arXiv preprint arXiv:2006.05523 2020.
[DV13] Guiseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. Proc. IJCAI 2013, pages 854-860, 2013.
[FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[GA14] Iiche Georgievski and Marco Aiello. An overview of hierarchical task network planning. arXiv preprint arXiv: TeXXiv::43...0 /403.7426, 2014.
[GB13] Hector Geffner and Blai Bonet. A concise introduction to models and methods for automated planning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 8(1):1-141, 2013.
[GNT04] Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated planning theory and practice. Elsevier, 2004.
[HBB+19 19] Daniel Höller, Gregor Behnke, Pascal Bercher, Susanne Biundo, Humbert Fiorino, Damien Pellier. and Ron Alford. HDDL A language to describe hierarchical planning problems. In Second ICAPS Workshop on Hierarchical Planning. 2019.
(Hel06) Malte Helmert. The fast downward planning system. J. Artif. Intell. Res., 26:191-246. 2006.
[HGDG17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pages 1980-2988 IEEE Computer Society, 2017.
[HLMM19] Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, and Christian Muise. An Introduc- tion to the Planning Domain Definition Language. Morgan & Claypool, 2019.
[HofO1] Jörg Hoffmann. FF: the fast-forward planning system. A/ Mag., 22(3):57-62, 2001. 2001.
[HPBL233] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillierap. Mastering diverse domains through world models. arXiv preprint 201.04230,.04144 2023.
11

---

[iBC
brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexan- der Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang. Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang- Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jau- regui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as i can, not i say: Grounding language in robotic affordances. Karen Liu, Dana Kulic, and Jeff Ichnowski, editors. Proceedings of The 6th Confer- ence Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 287-318. PMLR, 14-18 Dec 2023.
[1O22] Yuki Inoue and Hiroki Ohashi. Prompter: Utilizing large language model prompting for data efficient embodied instruction following. arXiv preprint urXiv:2211.03267, 2022.
[JLZ*22] Zhiwei Jia, Kaixiang Lin, Yizhou Zhao, Qiaozi Gao, Govind Thattai, and Gaurav Sukhatme. Learning to act with affordance-awee aware multimodal neural SLAM. In International Conference on Learning Representations, 2022.
[KWW95] Nicholas Kushmerick. Steve Hanks, and Daniel S. Weld. An algorithm for probabilissic planning. Artif. Intell., 76(1-2):239-28)) 1995.
[KMG+17] Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: an interactive 3d environment for visual AI. CoRR,
abs/1712.05474,,22 2017.
[KSV+22] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In Proceedings of the AAAI Conference Artificial Intelligence, pages 12262-12267, 2022.
[LCQ 18] Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi- Fu, and Pheng-Ann Heng. H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from CT volumes. IEEE Trans. Medical Imaging, 37(12):2663-2777 2018.
[LSS+222 Leonardo Lamanna, Luciano Serafini, Alessandro Saetti, Alfonso Gerevini, and Paolo Traverso. Online Grounding of Symbolic Planning Domains in Unknown Environments. In Proceedings of the 19th International Conference on Principles of Knowledge Representation and Reasoning, pages 51 511-521, 8 2022.
[MBM14] Christian J. Muise, Vaishak Belle, and Sheila A. Mcllraith. Computing contingent plans via fully observable non-deterministic planning. In AAAI, pages 2322-2329. AAAI Press, 2014.
[MBP*21] Shervin Minaee. Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[MC22] Michael Murray and Maya Cakmak, Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment. IEEE Robotics Autom. Lett. 733)66870-6877, 2022.
[MCR*20 So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. FILM: following instructions in language with modular methods. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[MCRW09] TL McCluskey, SN Cresswell, NE Richardson, and MM West. Automated acquisition of action knowledge. /st International Conference on Agents and Artificial Intelligence, pages 93-100. SeiTePress, 2009.
12

---

[MH13]
Christoph Mies and Joachim Hertzberg. Encoding hin heuristics in pddl planning instances. In KI 2013: Advances in Artificial Intelligence: 36th Annual German Conference on AI, Koblenz, Germany, September 16-20, 2013. Proceedings 36, pages 292-295. Springer, 2013.
[MK12] Mausam and Andrey Kolobov. Planning with Markov Decision Processes: An Al Perspective. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2012.
[MMB14 14] Christian Muise, Sheila A. Mcllraith, and Vaishak Belle. Non-deterministic planning with conditional effects. In Proc. ICAPS 2014, pages 370-374, 2014.
[MvSE22] Jorge A Mendez, Harm Seijen, and ERIC BATON. Modular lifelong reinforcement learning via neural composition. In International Conference on Learning Representa- tions, 2022.
[N]]+03] Dana S. Nau, Tsz-Chiu Au, Okhtay Iighami, Ugur Kuter, J. William Murdock, Dan Wu, and Fusun Yaman. SHOP2: An HTN planning system. Journal of Artificcal Intelligence Research, 20:379-404, 2003.
[ORCC21] Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and I Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298-30312, 2021.
[PG09] Hector Palacios and Hector Geffner. Compiling uncertainty away in conformant planning problems with bounded width. Journal of Artificial Intelligence Research, 35:623-675, 2009.
[PMM+22] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv preprint arXiv:2212.08681, 2022.
[SAH+20] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
[SB09] Scott Sanner and Craig Boutilier. Practical solution techniques for first-order mdps. Artificial Intelligence, 173(5-6):748 788, 2009.
[S] Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied BERT: A transformer model for embodied, language-guidee visual task completion. CoRR, abs/2108.04922 04927, 2021.
[SSS+ 17] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676) 359, 2017.
[0] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED; A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, pages 10737-10746. Computer Vision Foundation / IEEE, 2020.
[SWW+23] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models, 2023.
[TGD04] Prasad Tadepalli, Robert Givan, and Kurt Driessens. Relational reinforcement learning: An overview. Proceedings of the ICML--2004 workshop on relational reinforcement learning, pages 1-9, 2004.
13

---

[VSM*2]
Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Sub- barao Kambhampati. On the planning abilities of large language models (a critical investigation with proposed benchmark). arXiv preprini arXiv:2302.0677 06706, 2023. Previous versions accepted "Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022" with title "Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change)".
[WDKM21] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[WJK08] Chenggang Wang, Saket Joshi, and Roni Khardon. First order decision diagrams for relational mdps. Journal of Artificial Intelligence Research, 01::41144722 2008
[WK10] Chenggang Wang and Roni Khardon, Relational partially observable mdps. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, pages 1153-1158, 2010.
[WPY05] Jason D Williams, Pascal Poupart, and Steve Young. markov decision processes for dialogue management. In Proc. IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 76-82. Citeseer, 2005.
Factored partially observable
[YFG07] Sungwook Yoon, Alan Fern, and Robert Givan. Fe-Replan: A baseline for probabilistic planning. In Proc. ICAPS 2007, pages 352-360, 2007.
[2RT+22] Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Olek- sandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang. Alexander Clegg, Manolis Savva, Eric Undersander, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2022. https: https://aihabittat orr hwt.bb/aaas.og/gaaaaagg/////// challeege///22/////////// 2022.
[YZY+23] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.
[ZC21] Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self hifl-monitoiing In ACLIICCLPP (Findings), volume ACL/I- JCNLP 2021 of Findings of ACL, pages 4202 4213. Association for Computational Linguistics, 2021.
[2YP+22] Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma. Keun- woo Peter Yu, Yuwei Bao, and Joyce Chai. DANLI: deliberative agent for following natural language instructions. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1280-1298, Association for Computational Linguistics, 2022.
14

---

A Supplementary Material
This Supplementary Material section provides additional details and discussions to complement our main paper. In section B, we expand on the discussion of our approach. Section C presents specific results for the seven tasks of the ALFRED challenge as well as new tasks that can be solved in the environment. Section [ariion[[pr provides a formal definition of Object-Orienter Partially Observable Markov Decision Processes (OO-POMDPs), while section provides a detailed background for our planning approach. Finally, section F provides in-depth explanation of our method for the ALFRED challenge.
B Further Discussion
In section 9 we touched on various aspects of our methodology. This section provides more detail on these areas. Our approach to the ALFRED challenge hinges on egocentric planning and user-defined symbolic planning model. We refer to these models as intentional, as they are purposefully crafted by the user, integrating specific environmental objects and properties. Symbolic models function as a form of world model that facilitates planning. Some methods seek to learn latent representation, but this often comes with a high sample complexity (HPBL23).. Modular meta-learning can reduce the sample complexity (ALPKI8 ORCC21 MvSE22). If cost is not a constraint, world models can become components of efficient strategies trained to solve problems in specific domains (SAH+ 20).
However, with limited resources or data, generalizing to new situations can be challenging. Take an embodied agent in an environment (.8) with N skills and M types of objects, tasked with carrying out a sequence of tasks that require plans involving a new combination of k skills and m types of objects, where n < N and m < M Such setting is within the scope of methods like meta- learning or continual learning for adapting the latent representation for such unfamiliar environments (FAL17). The priors in some modular meta-learning approaches are related with our notions of objects and actions, but their inference algorithms might not scale well for long plans or ignore that some users might also want intentional modelling (ALPKI8) [RCC21 MvSE22). Egocentric plamning, the other hand, may provide predictable scalability in these scenarios. While the definition of open-world generalization might be more clusive, our method leans towards systematic generalization across actions and object types.
Planning using certain latent representations or world models can also prove costly. As explained by (AKFM22) (LATPLAN), not all factored representations afford the same level of scalability. The authors of LATPLAN apply variational autoencoders to learn a discrete latent representation, incorporating additional priors so the representation can be interpreted as a STRIPS symbolic model, the most basic type of symbolic model we use for egocentric planning. Symbolic models like STRIPS enable solvers with greater scalability and generalization.
An alternative approach involves methods that learn symbolic planning model (AFP+ 18 CDVA+22). Object-oriented POMDPs are especially suitable as the grounding on objects and action labels allows for more efficient algorithms (CMWI3 MCRW09), an advantage not enjoyed by the authors of LATPLAN as they only use pairs of images as training data. Learning a symbolic model helps manage the risk of compounding errors since the model can be debugged and improved. Real-world applications may require monitoring the agent's 's behaviour for specific objects or actions. Furthermore, specific actions may carry inherent risks that need to be avoided. In general, we antici- pate that developers of embodied agents will use as much learning as possible, as much intentional modelling as necessary, and symbols to blend the two of them (S)
Further integration between symbolic planning models and other ML models might help to reduce some limitations of our approach. For instance, in ALFRED, searching for a pan should make the agent head toward the kitchen. If another model like an LLM finds it likely that the pan is in the kitchen, we can use the scores to produce action cost a similar way to the integration done in SayCan, 23).
Egocentric planning connects body of work aimed at tackling expressive planning problems by harnessing the scalability of classical planners. While egocentric planning focuses on the partial observability of objects, (APG09 PG09) consider partial observability given set of possible initial
While the state space is unknown to the agent, it is aware of the actions and object types.
15

---

states and deterministic actions, and (MMB14) considers full observability but with nondeterministic actions. Though these strategies can provide complete solutions proven offline, they can also serve as approximate methods for solving online problems. When planning with probabilistic effects, replanning assuming randomized determinization of actions can be strong approach there are no dead ends (YFG07).
Our symbolic planning models are extensions of STRIPS, but other options exist. Hierarchical task networks (HTN) (NAI 03 GAT4), for example, offers an alternative to STRIPS as they allow the specification of plans at various abstraction levels. Sometimes, it is more practical to outline potential plans using formalisms like linear temporal logic (LTL) formulae (DV13). Nonetheless, the relationship between these models has been examined, and it is feasible to integrate them with STRIPS- baased planning (CC04 HBB 19 MHI3).
C Results in ALFRED's Task and New Tasks
C.1 ALFRED Task Performance
ALFRED Tasks
Valid Unseen
Valid Seen
SR
GC
SR
GC
look at ot_obj__ in light
53.77
54.27
47.22
30.11
bick_and_place_siin recep
43.14
47.11
37.15
39.77
46.76
45.82
47.01
48.89
pick_clean_ttore
faatt_tt__llae_iing palace_in_recep recep
38.77
41.37
38.01
42.21
lit__coo_tthe__pllii jeccepppppeeepppp
46.26
46.81
47.21
47.33
wtw_hheat_then_pace_e___
37.69
40.50
39.82
41.33
30.33
35.71
27.37
31.21
Table 5: Performance on Valid Unseen and Valid Seen for each task
To analyze the strengths and weaknesses of our approach in different types of tasks, we conducted further evaluations by task type as in Table Our findings reveal several noteworthy observations. Firstly, the success rates (SR) and goal completion rates (GC) for the "Pick two (wo_obj___ ho__oj__anndd_ and place" task are lower than the rest, mostly due to these tasks involving more objects and with smaller masks. The task "look obj in light" has the best performance due to the variety of lamp is low compare to other common objects. Secondly, there is no strong correlation between the number of interactions involved in task type and its performance 2S) For example, the task "pick_ _col__ooo_____ooeeooeee Lhol__hee_____ theetteeehheee Bacpppplcee____ee in receptacle" has an extra cooling step compare to "pick_and Loed__laee_sinne but still have better performance. detailed description of each task can be found in the original ALFRED paper on page 16 to 19 (SGG 20).
C.2
New Task Performance
In section
8.4
we discuss the generalization of our approach to new tasks. The results are summarized
table
SR
GC
PLWSR
PLWGC
Pickup and Place Two Objects
90.00
93.44
3.71
4.37
Clean-and Ceean-and-Heal Objects
80.00
89.77
2.56
3.90
Clean-and-Con Objects
85.00
93.38
4.44
4.78
Heat-and-Cou Objects
80.00
89.20
4.01
4.66
Pick and Place Object in Drawer
70.00
77.81
2.59
3.22
Table 6: New tasks
Our approach enable us to reuse the definition of existing models to achieve zero-shot generalization to new tasks as long as the set of actions, objects and their relationships remain the same. In many embodied agent applications, agent need to have ability to adapt to new tasks. For instance, organizing living room could involve storing loose items a shelf or arranging them on table. An
16

---

agent familiar with common household objects and equipped with the skills to handle them should seamlessly manage both types of cleaning tasks. However, transfer learning for neural networks remains challenging in the context of embodied agent tasks generalizing across objects and skills. Similarly, the template system used in FMLM demands hand-crafted policies for new task types. The ALFRED dataset include instructions for each tasks that (STG+200) although our approach does not use them. In a our egocentric planning formulation, a task can be broken down into a set of goal conditions that comprise objects of interest. The planner can then autonomously generate an action sequence to reach these goals. Consequently, our method can independently execute these tasks without the need for transfer learning or the addition of new task templates. To demonstrate this, devised 5 new task types that were not present in the training data. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made to minimize perception errors. These tasks share the same world model as ALFRED and thus do not necessitate any additional modification beyond specifying a new goal. We selected environments and layouts from the training set where subgoals required for each new task type are feasible, ensuring the feasibility of the new task for agent. Table lists these new tasks and their success rates. We selected 20 scenarios for each new task and maintained the maximum 10 interaction error as in the ALFRED problem. Our method was able achieve an impressive overall success rate of 82% without any additional adjustments beyond the new goal specification. However, due to the constraints of the ALFRED simulator, we 'e unable to experiment with introducing new objects and action types beyond what's available. For instance, "heating" in ALFRED is solely accomplished via microwave. In real-world scenarios, we could easily incorporate ovens and actions to operate them with minor alterations to our PDDL domains. Similarly, we could introduce time and resource restrictions for each action, which are common in many PDDL benchmarks. The ability to accommodate new goals and constraints is a key advantage of our planning- based approach.
C.3 Computation
Perception and Language module was fine-tuned on Nvidia 3080. The planning algorithm uses 8 core Intel i7 CPU with 32 GB of ram. More detailed on traning can be found in (MCR+22)
D Embodied Agents Generalizing within given Objects Types
In this section, we provide further details about embodied agents presented in section 4 Embodied agents perceive and act in environments to accomplish tasks. Navigating challenges and constraints imposed by their environment and tasks, these agents ensure safety and adhere to physical restrictions. Taskable agents rely their surroundings and inherent capabilities, creating strong dependency and often specializing in particular environments. This specialization shapes their cognition and goblemm-olliig strategies, presenting an opportunity for robust agent design.
We exploit this limitation by focusing on generalization across tasks involving the same object types and their relationships, applicable to both physical and software agents. Taskable agents possess specialized capabilities for these object types, as seen in robotics where battery level, body state, and actuators are distinct variables (CST+22). Object types also serve as inputs/outputs in end-to- end approaches with learned representations and appear in embodied agents operating in software environments, using APIs and maintaining internal states. Our symbolic planning approach aims to provide basis for designing adaptable and flexible agents that sense and act upon given object types. We start by defining object types, and entities grounded in these types.
Definition D.1 (Objects and Object Types). An Object Type, denoted as (T, V)), consist of a finite set of object types T, along with a mapping V t2 - V; that assigns each type 4; E T a set of possible values V,. An Object o = (ts, v) is an instance of a type with a specific value, i.e., 4, T and (tt)..
Example D.2 (Object types and Values). An example of an object type tobjectSeen with values V (1aa) x Vdirection x VobjectSubtype x Via where Vaa.a) is position in a grid, Vdirection is the direction the object is facing, VebjectSubtype is the category of the object observed, and Via is a unique identifier for the object. Other reluied examples of object types are monablee with values V VmovabieS x V.ax V Vobjectproperties and lobsttatte with values VehstacteSubtupe x Ves x Vohjectgroperties- Furthermore, an action p ckup with parameters &e (a) = (fobjectSeen) is action that picks up specific object.
17

---

Object types are used as parameters of entities of actions of Object-oriented POMDP defined, and as parameters of actions and predicates in planning domains in section E
Definition D.3 (Typed Entity). Given set of entities E, object types (T, and a mapping
: UteN Th. A Type Entity & = is mapped to (5) (41, 12..,, (kg),, with each 44 E T. This mapping associates each entity with a kg-tuple of object types, corresponding to lgg parameters of the entity, defining the arity of the entity &. We refer to the variables var E {vari, var2, ... vark. as the positional parameters of the entity ki, denoted E(var1, ......... ...,, wark).
Definition D.4 (Grounded Entity). Given a set of typed entities E. with object types (T. V), and mapping 4 a Grounded Entity, denoted as Ç for & E E, is parametric object where each type argument of & specified by ((E) is replaced by an object of the corresponding type. Specifically, <" { o2...., Okg), where each o (t4, 24) with U; E (t+)). The set of all grounded entities is denoted by 3
Now we define the environment where the embodied agent operates as Partially Observable Markov Decision Process (POMDP) with costs. Cost POMDPs are related to classical MDPs but feature partial observability and use costs instead of rewards (MK12 GB13).
The literature sometimes confound the environment with the underlying the POMDP, but we want to make explicit what is available to the agent. Thus, while our underlying model is POMDP, we consider agents that can aim to generalize without being fully aware of the distribution of future tasks, except for the actions and object types that are explicit in the environment. Therefore, we abuse slightly the notation to annotate with the objects defined below with the environment, denoted E.
Definition D.5 (Cost POMDP). A Cost POMDP is tuple (Se, Ag, Te. Ce. Pe, Zg), where Sg is a set of states, Ag is a finite set of actions, T& Sg x Ag x Sg + [o, 1] is the transition function, Cg: : Sg Ag + R >> 0 is the cost function, Pe is of perceptions, and Ze X [0, 1] is the sensor function.
Object-oriented POMDP is an extension of a Cost POMDP that includes objects and their properties to be used for expressing perceptions and actions.
Definition D.6 (Orject-oriented POMDP). An Object-oriented POMDP r, represented by (Se, Ae, &g, Te, Ce,PE, Ze, Te, VE), extends Cost POMDP by incorporating a predefined set of Object Types (Te, Vε). It defines the set of perceptions Pe where a perception p E Pg is a finite set of objects {0r, .2,..., 0m.) and the map &g--
and the set of typed actions Ag defined for the object types (Te, VE)
In this context, Grounded Actions refer to grounded entities, where the set of entities is the actions Ag. Values in object-oriented POMDPs can be dense signals like images or sparse signals like object properties. Values can have any structure, so they can refer to values in the domain of other types. Indeed, Object-oriented POMDPs can express Relational POMDPs (WKI0 WJK08 SB09 TGD04] WPY05 DNR 20). However, our approach emphasizes the compositionality of the objects and their propertics.
Agents interact with the POMDP through an environment. Given a task, we initialize the environment using the task details. The agent receives the tasks and acts until it achieves the goal or fails due to some constraint on the environment. The agent achieves a goal if, after sequence of actions, the environment's 's hidden state satisfies the goal.
Definition D.7 (Task). Given Object-oriented POMDP r, Task is tuple Tg (Ig, Ge), where Ig is an object that the environment interpret for setting the initial state, and Ge is a set of goal conditions expressed objects {g1, ... Gn with each gi being an instance of an object.
Definition D.8 (Object-oriented Environment). Given a hidden Object-oriented POMDP I, an Object- oriented Environment, or just Environment, is a tuple = (As. Te. VE, reset, step). where Ag, Te. and Ve are as previously defined, reset: : Ts Pe is a function that takes task T3 (Ie, Gæ), resets the environment's internal state using 1& and returns an initial observation, and step A + Pe x IR is the interaction function that updates the environment's 's internal state according to the execution of a grounded action ag. for E Ag, and returns an observation and a cost.
In some environments, Te might be opaque for the agent. In others, it might containg information not used by the environment like user's 's preferences about how to complete task. However, we abuse this notion and refer to Ig as the initial state of the environment.
18

---

Separating Object-orienter POMDPs from environments allows us to consider the scope of the generalization of our agents within the same POMDP. While the agent cannot see the state space directly, it is aware of the actions and the object types.
Definition D.9 (Embodied Agent). Given an environment & == (Ag, Te, Vε, reset, step). and task Te = (Ig, Gg), an Embodied Agent is a tuple (M TM M GM where M, Pe x Ig x Gg - M a function that takes initial observation, a task, and returns the initial internal state of the agent, TM : .M- + A is the agent's policy function that maps mental states to a grounded action in Ag. My .M Ag PE x Ik- + M is the mental state update function receiving grounded executed action, the resulting observation. and the cost, and Gy M - [0, 1] is the goal achievement belief function that returns the belief that the agent has achieved the goal based on its mental state M.
Algorithm provides general framework for an agent's s execution process. By instantiat- ing the policy, mental state update function, and goal achievement belief function using appropriate techniques, the agent can adapt to a wide range of environments and tasks. Although new tasks can set the agent in un- seen regions of the state space Sg, the agent remains grounded on the known types and the actions parameterized on object types, Te- They offer opportunities for generalization by narrowing down the space of possible policies and enabling compositionality. The agent's s mental state M can include its belief state, internal knowledge, or other structures used to make decisions. The policy TM can be a deterministic or stochastic function, learned
Algorithm 2 Agent Execution
Input: Environment (Ag, Te, VE, reset, step) Input: Agent (M,, TM My. Gm)
Input: Task (Ig, Gg} Input: Probability threshold 0 Output: Successful trace 7 or Failure
1: p reset(Te, G&)
2: T = I
> Empty trace
3: M t- (p Gg)
4: while GM < 0
5:
a =a(01, 02, ..., OK Tas (M, Ag)
D where a in Ag and of are objects. aappeed((a)) p', c step(a")
6:
7:
8:
if failed E p' then
return Failure 10: M M M (M, a² cc
9:
from data or user-specified, or combination of both. As the agent has an internal state, its policy can rely on a compact state or on the history of interactions.
11: return
The agent starts the initial state set from Ie and aims to achieve the goal Gg. While the agent does not know the true state of the environment, agents can aim to generalize across actions and the object types as they are always available. While we prefer policies with lower expected cost, our main concern is achieving the goal. It is possible that a task might be vnsolv possibly because of actions taken by the agent. We assume that failures can be detected, for instance, by receiving a special observation type failed. We further assume that after failure, costs are infinite, and failed is always observed.
The Iterative Exploration Replanning (IER), Algorithm 1 in section 5 is instance of algorithm 2 Instead of using a parameter 6, it considers the problem solved when it obtains a plan that achieves the goal.
E Detailed Background: Parameterized Full-Observable Symbolic Planning
In this section, we describe the background definitions of parameterized full-observable symbolic models mentioned in section 5. We define planning domains and problems assuming full ob- servability, including classical planning that further assumes deterministic actions, also known as STRIPS (GNT04) GB13). Planning domains and problems rely on typed objects (Def D.I). as object-oriented POMDPs, but planning actions include model of the preconditions and effects. Definition E.1 (Parametric Full-Observable Planning Domain). A parametric full-observable plan- ning domain is tuple PD = (T, V. P. .A. b), where (T, V\) is a set of object types, P is set of typed predicates and AA is set of typed actions, both typed by the object types and the map 4. Each action a has an associated PRE,, expressing the preconditions of the action, and EFF expressing its effects. For each grounded action a(o), 02 ..., oR) E A9, the applicability of the grounded action in a state s is determined by checking whether the precondition PRE. is satisfied in The resulting state after applying the grounded action is obtained by updating 8 according to the effect update EFF.- The specification of EFF, is described below.
19

---

While the values in both parametric full-observable planning domains and object- deject--riented POMDPs can be infinite (d) the values used in concrete plamning problems are assumed to be finite. As a consequence, the set of grounded actions of planning problems is finite (def
Definition E.2 (Parametric Full-Observable Planning Problem). A parametric full-observable plan- ning problem is a tuple (PD, O, G), where PD = V, V, P, A,, Φ) is parametrie full-observable planning domain, O a finite set of objects, each associated with type T, is a description of the initial state specifying the truth value of grounded predicates in P9, G is description of the goal specifying the desired truth values of grounded predicates in ps, where po are the predicates P grounded on object types (T, V) and the map & encoding their typed arguments.
This definition covers wide range of symbolic planning problems featuring full observability including probabilistic effects (KHW95). In particular, the definition can be specialized for classical planning problems, where the actions are deterministic, corresponding to STRIPS, to the most studied symbolic model (GB13).
For classical planning, given an action a E A parameterized by list of object types a(t2, b2,,..,, (2), their precondition or effect are formulas 7 expresse in terms of the variables var E {var1, 2,..,..., vark) that refer to positional parameters of the action. We denote the grounded formula 7(11 02...., ox) as y with each variable var; that occurs in y is replaced by corresponding objects o,.
Definition E.3 (Parametric Classical Planning Domain). A parametric classical planning domain is a parametric full-observable planning domain where each action a E AA has preconditions PRE., and an effect update EFF, represented as a tuple (ADD,, DELa) of add and delete effects, all sets of predicates in P. Given a grounded action a ((Orry) with Oary = 01, og,..., Ob, the precondition PRE, satisfied in state if PREgg(orgg) S s. The effect update EFF, describes the resulting state as (s DEL (Porg)) )) U ADD,, after applying the action.
Definition E.4 (Parametric Classical Planning Problem). Given parametric classical planning domain PD, a parametric classical planning problem is a parametric full-observable planning problem where and G are conjunctions of predicates in P (equivalently viewed as sets).
As Egocentric Planning relies on replanning over a sequence of related planning problems, it is convenient to define some operations required for exploration, as the agent discovers more objects.
They allow us implement the function ExploreAct in Algorithm [
Definition E.5 (Precondition Extension). Given a parametric full-observable planning problem and an action a E A, extending the precondition PRE. with APRE,, denoted PRE., APRE,, is new precondition that is satisfied in state s if PRF., and A PRE. are both satisfied in s.
Definition E.6 (Effect Extension). Given a parametric full-observable planning problem and action a E A, extending the effect EFF, with (AADD,, ADELa). denoted EFF, @ (AADDa. ADEL.) ). is a new effect that applied to state results the new state s' that is like but with modifications E then ADEL, is not true in s', and ÅADD, is true in s', in that order, assuming that neither ADEL, nor ADDD, appear in EFF..
With these operations we can define the Egocentric Planning algorithm for parametric full-observable planning problems. For instance, in the case of probabilistic planning the preconditions can be defined as in classical planning, so the precondition update is the same. The effect update of probabilistic actions is probability distribution over the possible effects. However, extending the effect with a new effect straightforward as the new effect must enforce that Pr(ADD.|s) = 1 and (r (s) 0.
For classical planning, the precondition extension is the union of the original precondition and the new precondition both are set of predicates. The EFF, of classical planning actions (ADD,,. DEL.) is a tuple of sets of predicates, so updating the effect amounts to updating both sets of predicates. Definition E.7 (Precondition Extension Classical). For parametric classical planning problems, extending the precondition PRE, @ APRE, is PRE, U APRE,
Definition E.8 (Effect Update Classical), For parametric classical plamning problems, an update to the effect (ADD,, DEL.) with (AADD,, ADEL.) (ADD, AADD,, DEL, U ADEL.).
For ALFRED, as the planning domain in classical, ExploreAct (a, o) creates a copy of a extended with new precondition (unknown o) so can only be applied if o is still unknown. The effects are extended by deleting (unknown o) and adding (explored), so applying the exploration action satisfies the goal (explored).
20

---

then provide Approach: Additional Details
In this section, we provide additional details on our approach discussed section 6 Both the environment and the symbolic planning model consider objects and parametric actions, but they represented differently.
For instance, our approach involves a vision model that perceives in scene objects of known classes, such as apple or fridge, including their depths and locations (see sections 4 and D). Similarly, the planning domain includes corresponding object types, such as AppleType and FpidgeType (see sections 5.T and Ep. When the agent perceives an apple at location 11, the algorithm creates fresh object o1 for the apple, sets its type to (objectType o1 AppleType), and its location to o1 11). Additionally, the domain contains information about what can be done with the apple, such (canCool FridgeType AppleType). Likewise, while the ALFRED environment contains actions like toggle, the planning domain contains actions like cool0bject, arguments such the location, the concrete object be cooled, and
which is parameterized the concrete fridge object. Finally, if the task provided in natural language requests to cool down apple, the planning problem goal contains, for instance, (exists (isCooled led ?o)), which is satisfied when some object has been cooled down.
The remainder of this section is structured as follows. First, we provide additional details on the Language and Vision modules, including the specific classes they detect and how they are utilized. Next, we elaborate on the semantic spatial graph and its role in generating the agent's mental model, which tracks object locations as the agent perceives and modifies the environment. specific details the planning domain used for ALFRED, which completes the elements used in our architecture depicted Figure Finally, we conclude the section with additional details on the egocentric planning agent that won the ALFRED challenge.
F.1
Language and Vision Module
Table 7: List of Objects
alarmclock
apple
armchair bed box
baseballbat blinds bread
basketball
bathtub
bathtubbasin
book
boots
bowl
butterknife
cabinet
candle chair cloth
cart
cd
cellphone
coffeemachine countertop
creditcard
cup
curtains
desk
desklamp
dishsponge
drawer
dresser
cgg
floorlamp
footstool
fork
fridge
garbagecan
glassbottle
handtowel
handtowelholder
houseplant
kettle
keychain
knife
ladle
laptop
laundryhamper
laundryhamperlid
lettuce
lightswitch
microwave
mirror
mug
newspaper
ottoman
painting
pan
papertowel
papertowelroll
pen
pencil
peppershaker
pillow
plate
plunger
poster
pot
potato
remotecontrol
safe
saltshaker
scrubbrush
shelf
showerdoor
showerglass
sink
sinkbasin
soapbar
soapbottle
sofa
spatula
spoon
spraybottle
statue
stoveburner
stoveknob
diningtable tissuebox
coffeetable
sidetableteddyyear
television
tennisracket
toaster
toilet
toiletpaper
toiletpaperhanger
toiletpaperroll
tomato
towel
towelholder
tvstand
vase
watch
wateringcan
window
winebottle
F111 Language Module
We used a BERT-based language models to convert structured sentence into goal conditions for PDDL (DCLT19). Since the task types and object types of defined in the ALFRED metadata, we model to classify the type of task given high-level language task description listed in Table 9
21

---

Table List of Receptacles
Bathtub
Bowl
Cup
Drawer
Mug
Plate
Shelf
Sink
Box
Cabinet
CoffceMachine
CounterTop
Fridge
GarbageCan
HandTowelHolder
Microwave
PaintingHanger
Pan
Pot
StoveBurner
Dining Table
CoffeeTable
Side'Table
ToiletPaperHanger
TowelHolder
Safe
BathtubBasin
ArmChair
Toilet
Sofa
Ottoman
Dresser
LaundryHamper
Desk
Bed
Cart
TVStand
Toaster
Table 9: List of Goals
Lii__anndplace_singe look_at "___at_obj_in__ _obj_inllight
wick_two_obj_and_plar
bactlean_ttee_paaeee_ jeneee____eeepppppp
pick_heat_theer teceatt_teen_paseee_e_ thtet_ttaceii_neeeeeee
lacanc_ii_rreee in_ep
pick_ani hand_place_________ with movable recep
and a separate model for deciding objects and their state based on the output of the second model in Table 7 and Table [8] For example, given the task description of "put apple on the kitchen table," the first model will tell the agent it needs to produce a "pick-up-and-place task. The second model will then predict the objects and receptacle we need to manipulate. Then the goal condition will be extracted by Tarski and turn in to a PDDL goal like (on apple table) using the original task description and output of the both models. We use the pre-trained model provided by FILM and convert the template-based based result into PDDL goal conditions suitable for our planner. Although ALFRED provides step-by-step instructions, we only use the top-level task description since they are sufficient to predicate our goal condition. Also, our agent needs to act based on the policy planner, which makes step-by-step instructions unnecessary. This adjustment could also make the agent more autonomous since step-by-ster instructions are often not available in many embodied agent settings.
E.1.2
Vision Module
The vision module is used to ground relevant object information into symbols that can be processed by our planning module. At each time step, the module receives egocentric image of the environment. The image is then processed into a depth map using U-Net, and object maks using Mask-RCNN, (HGDG17 LCQ 18). Both masks are WxH matrices where W, and N are the width and height of the input image. Each point in the depth mask represents the predicted pixel distance between the agent and the scene. The average depth each object is calculated using the average of the sum of the point-wise product of the object and the depth mask. Only objects with average reachable distance smaller than meter is stored. The confidence score of which our agent can act on an object is calculated using the sum of their mask. This score gives our egocentric planner a way to prioritize which object to interact with.
F.2 Semantic Spatial Graph
The spatial graph plays the role of agent's 's memory during exploration and bridges the gap between grounded objects and the planner's state representation requirements. The exploration employs an egocentric agent who only has direct access to its local space where action can be executed within a distance threshold of 1.5 meters. The key for each node is encoded as location, with visual observations being the values. The edges represent the agent's 's actions, comprising four specific actions: MOVEAHEAD, MOVEBACK, TURNLEFT, and TURNRIGHT, with movement actions having a step size of 0.25 meter and turning actions featuring a turn angle of 90". This spatial graph is updated and expanded over time through policies generated by our egocentric planner, with each node storing object classes, segmentation masks, and depth information produced via the vision module. Fig. 2 shows an example of semantic spatial graph.
22

---

Non-Object Nodes
Object Nodes
Figure 2: Example of Semantic Spatial Graph after partial Exploration. The agent one of the nodes.
The spatial graph undergoes continuous updates and expansions result of the policies generated by our egocentric planner. It involves the extraction and storage objects within the graph, subject to confidence score threshold of 0.8. Objects that meet exceed this confidence score deemed reliable and considered for storage. Each node within the graph maintains record of various attributes associated with the objects, including object classes, segmentation masks, and depth information. The object classes provide information about the category or type of each object, allowing for efficient categorization and analysis. The segmentation masks of the extracted objects are stored list of pixel coordinates. These masks outline the boundaries of each object within the captured visual data. By storing the masks in this format, can later retrieve and utilize the spatial information of each object during subsequent analyses or interactions. Moreover, the distance to each object is calculated by leveraging the predicted distance of the top 100 pixels with reference to the depth mask generated by the U-net model. The U-net model is responsible for producing depth mask that estimates the distances various points within the scene. By employing the predicted distance values of the top 100 pixels associated with each object, we obtain an approximate measure of its distance from the egocentric viewpoint. Actionable objects are treated as affordances at each node, allowing the agent to perform desired actions. Nodes with objects of interest are ranked based on the confidence scores outlined earlier output by the MaskRCNN.
F33 Egocentric Algorithm for ALFRED
The egocentric planning algorithm for ALFRED uses location as the only anchor types T., and the move action the only exploration action X. Consequently, exploration plans employ move actions to visit unexplored locations, revealing new objects visible from such locations. As the agent explores, the mental model creates new objects and adds them the initial state, facilitating the incremental construction of a semantic spatial graph, which maps the environment and object locations. An ALFRED task, Tr (Iz, Ge), characterized by goal that corresponds to one of the seven tasks detailed in section 2 and involves specific predicates for the task-related objects. The goal is expressed as an existential formula over potential objects, as it refers objects not visible from the initial state (Listings
23

---

We also implemented some ALFRED SEFRED-specific optimizations. For example. we mitigate the risk of executing irreversible actions by only attempting them after devising a plan projected to achieve the goal. This strategy implies that some actions are unavailable during the exploration phase. For each time step, our egocentric planner will produce goal or exploration action depending on whether the agent has sufficient information to achieve its goal. The replanning loop repeats when an action fails. Priority are given to unknown location based on objects interest for particular task and number of total objects found in area. They are ranked base number of such objects. Information are gathered for each action taken which stored the semantic map. In order to reduce the amount of overall exploration required the online planning stage, allow agent first conduct 500 random exploration movements. The semantic graph after exploration is used as the initial state for our egocentric planning setup. To enhance efficiency, tailor the planning domain to the specific task, disregarding actions unlikely to contribute to reasonable plans to achieve the goal. This optimization be automatically implemented via reachability analysis, technique commonly used classical planning to prevent grounding actions irrelevant to achieving the goal from the initial state (Hof01).
F.4 PDDL Example
Below we provide examples assuming that the natural language description of the task is "heat potato in the microwave and put the potato on the table." We start by generating the planning setup the egocentric planner, given the task identified "pick- spick-heat-then tethee-placeii-reeeee involved in this manipulation are classified as "MicrowaveTType" "PotatoTyppe" and "Table'Type" The egocentric planner will eventually produce a sequence of actions to accomplish the desired task. In this case, the steps might include picking up the potato, heating it, and then placing it the table.
The objects
Listings 1 and show the PDDL domain and problem files for the example task. In PDDL domain for classical planning, both predicates and actions arguments are expressed as list of arguments, each one variable with an optional type. For instance, (conn ?x1 location ?x2 location ?x3 movement) express a predicate conn with three arguments: two locations and kind movement: ahead, rotate-left or rotate-right. In a PDDL problem for classical planning, object constants are associated with an optional type, and the initial state is a list of predicates grounded on existing constants. For instance, 11 13 location means three constants of type location. The initial state could be (and (at 11) (conn 11 12 ahead)). An action (MoveAgent ?from ?to location ?dir movement)) has as precondition that the agent and both locations are contiguous in the direction ?dir. Executing the action sets to false that the agent is at the location ?from, and sets to true that the agent the location ?to. Classical plans in PDDL sequences of actions grounded on constants. For instance, a first possible action might be (MoveAgent 11 12 ahead), leading to the state (and (at 12) (conn 11 12 ahead) ). For an introduction to PDDL see (HLMM19).
the location ?from
Listing I: ALFRED PDDL domain file
(define (domain alfred_ task)
(:requirements :equality :typing) (:types
agent object
location object
receptacle object
obj object
itentype object
movement object
12
(:predicates
13
(conn ?x1 location ?x2 location ?x3 - movement) (nove ?x1 movement)
14.
15
(atLocation ?x1 agent ?x2 location)
16
(receptacleAtlocation ?x1 receptacle ?x2 location)
17
(objectAtLocation ?x1
obj ?x2 location)
18
(canContain ?x1 itentype ?x2 itemtype)
19
(inReceptacle ?x1 obj ?x2 receptacle)
(recop[nneceepaclee ?x1 receptacle ?x2 receptacle)

---

(isInReceptacle ?x1 obj)
(openable ?x1 itemtype)
25
(canHeat ?x1 itemtype ?x2 itemtype)
24
(isHeated ?x1 obj)
25
(canCool ?x1
itemtype ?x2 itemtype)
(isCooled ?x1 obj)
(canClean ?x1 itemtype ?x2 itemtype)
28
(isCleaned ?x1 obj)
29
(iaMovableRcceptacle ?x1 receptacle)
30
(receptacleType ?x1 receptacle ?x2 itemtype)
(objectIype ?x1 obj ?x2 itemtype)
32
(holds ?x1
obj)
33
(holdsReceptacle ?x1 receptacle)
34
(holdsAny)
(unknown ?x1 location) (explore)
37
(canToggle ?x1 obj)
38
(isToggled ?x1 obj)
39
(can8lice ?x1 itemtype ?x2 itentype)
(isSliced ?x1 obj)
41
(emptyObj ?x1 obj)
42
(emptyReceptacle ?x1 receptacle)
43
(raction ssplore_MoveAgen.
46
:parameters (?agent0 agent ?from0 location ?loc0 location ?mov0 movement) :precondition (and (atLocation ?agento ?from0) (move ?movo) (conn ?from0 ?loc0 ?movo)
47
48
50
(unknown ?loc0) (not (explore))) reffect (and
51
52
(not (atLocation ?agentO ?from0))
(atLocation ?agentO ?loc0) (not (unknown ?loc0)) (explore))
56
57
(:action MoveAgent
:parameters (?agentO agent ?from0 location
60
?loc0 location ?oov0 movement) :precondition (and (atLocation ?agento ?from0)
61
(move ?movo) (conn ?from0 ?loc0 ?mov0)) :effect (and
(not (atLocation ?agentO ?from0))
65
(atLocation ?agent0 ?loc0))
66
(:action pickupObject
69
:parameters (?agentO - agent ?loc0 location ?objo obj) :precondition (and (atLocation ?agento ?loc0) (objectatlocation Tobj0 ?loc0)
70
21
(not (isInReceptacle ?obj0)) (not (holdsAny)))
73
:effect (and (not (objectAtLocation ?objo ?loc0))
74
75
(holds ?obj0) (holdsAny))
78
79
(:action pickupobjectFrom
80
:parameters (?agento agent ?loc0 location ?obj0 obj ?recepo receptacle) :precondition (and (atLocation ?agentO ?loc0)
85
(objectàtLocation ?obj0 ?loc0) (isInReceptacle ?obj0) (inReceptacle ?objo ?recepo) (not (holdsAny))) reffect (and
84
25

---

(not (objectAtLocation ?obj0 ?loc0)) (holds ?obj0)
88
(holdsAny)
89
(not (isInReceptacle ?obj0))
90
(not (inReceptacle ?obj0 ?recep0)))
95
(:action putonReceptacle
94
:parameters (?agentO agent ?loc0 location ?obj0 obj
95
Potyped itentype Precep0 receptacle
Prtype0 itentype)
97
:precondition (and
98
(atLocation ?agentO ?loc0)
99
CreceptacleAtLocatioo ?recep0 ?loc0)
(canContain Prtyped ?otype0) (objectType Pobj0 ?otype0) (receptacleType ?recep0 ?rtype0)
102
(holds ?obj0) (holdsAny) (not (openable ?rtype0))
103
(not (inReceptacle ?obj0 ?recep0)))
104
:effect (and
(not (holdsAny))
(not (holds ?obj0))
107
(isInReceptacle ?objo)
108
(inReceptacle ?objo ?recep0)
(objectAtlocation ?objo ?loc0))
111
112
(raction putinReceptacle
113
:parameters (?agentO agent ?loc0 location ?objo obj
?otyped itentype ?recep0 receptacle
?rtype0 itemtype)
116
:rrecondiiion (and
117
(atLocation ?agent0 ?loc0)
(xeceptacleAtLocatioon Precep0 ?loc0)
(canContain ?rtyped ?otype0) (objectIype Pobjo ?otype0)
(receptacleType Precepo ?rtype0) (holds ?obj0)
121
(holdsAny) (openable ?rtype0)
122
(not (inReceptacle ?obj0 ?recepo)))
:effect (and
(not (holdsAny))
125
(not (holds ?obj0))
126
(isInReceptacle ?obj0)
(inReceptacle ?obj0 ?recep0)
(objectAtLocation ?objo ?loc0))
130
(:action toggleOn
:parameters (?agentO agent Ploc0 location ?objo obj)
:precondition (and (atLocation ?agento ?loc0)
134
(objectAtLocation ?obj0 ?loc0) (not (isToggled ?objo))) :effect (and (isToggled ?objo))
135
139
(:action heatObject
140
:parameters (?agentO agent ?loc0 location ?obj0 obj
?otyped itentype ?recep0 receptacle ?rtype0 itentype)
145
:precondition (and
146
(atLocation ?agent0 ?loc0)
145
(recopracleALocattion ?recep0 ?loc0)
(canlleat ?rtype0 ?otype0)
(objectType ?objo ?otype0)
148
(receptacleType ?recepo ?rtype0)
149
(holds ?obj0) (holdsAny))
:effect (and
26

---

(isHeated ?objo))
155
Next, Listings shows the generated the planning problem based on the initial observation. This involves specifying the initial state of the environment and the goal that we want to achieve through the planning process.
Listing 2: ALFRED PDDL problem file with task achievement goal
(define (problem problem0) (:domain alfred_taski
(:objects
agent0 agent
AlarmClockType AppleType ArmChairType BackgroundType
BaseballBatType BasketBallType BathtubBasinType BathtubType
BedType BlindsType BookType BootsType BoulType BoxType
BreadType
ButterKnifeType CDType CabinetType CandleType CartType
10
CellPhoneType
iteettpe
11
f0_0_0f f0_0_1f f0_0_2f f0_0_3f f0_1_0f location Moveλhead RotateLeft RotateRight movement empty0 obj emptyR receptacle
12
14
15
16
(:init
(emptyObj empty0)
19
(emptyReceptacie emptyR)
20
(canContain BedType CellPhoneType)
(canContain CounterTopType PotatoType)
(canContain CoffeeTableType StatueType)
23
(canContain DiningTableType PanType)
24
25
(openable MicrovaveType)
(atLocation agent0 f0_0_0f)
(move RotateRight)
28
(move RotateLeft)
29
(move MoveAhead)
30
(canHeat MicrouaveType PlateType)
(canHeat MicrowaveTypp PotatoType)
32
(canHeat MicrowaveType MugType)
35
(canHeat MicrowaveType CupType)
34
(canHeat MicrowaveType BreadType)
(canHeat MicrowaveType TomatoType)
(canHeat MicrowaveType AppleType)
37
(canHeat MicrowaveType EggType)
38
(conn f0_0_1f f0_0_0f RotateLeft)
39
(conn f0_0_3f f0_0_2f
RotateLeft)
(conn f0_0_0f f0_0_1f
RotateRight)
(conn f0_0_1f f0_0_2f
RotateRight)
42
(conn f0_0_2f f0_0_3f
RotateRight)
43
(conn f0_0_0f f0.1_0f
MoveAhead)
44
(conn f0_0-2f f0_0-1f
RotateLeft)
(conn f0_0_3f 10_0_0f
RotateRight)
(conn f0_0_0f f0_0.3t
(unknown f0_1_0f) RotateLeft)
47
48
(:goal
(exists
52
(?goal0bj obj ?goalReceptacle receptacle)
53
(and (inReceptacle ?goal0bj "goalRecoptacle) (objectType ?goal0bj PotatoType)
27

---

(receptacleType ?goalReceptacle TableType) (isHeated ?goal0bj)))
57
58
As not plan was found for the problem in Listing the goal is changed to exploration in listing
Listing 3: ALFRED PDDL problem file with exploration goal
(: (:goal
(and (explore ) (not (holdsAny )))
28


Reviews:
[{"review_id": "lzclocCBMo", "reviewer": "Reviewer_oR6u", "summary": "The authors propose a hybrid approach leveraging neural perception models and symbolic planners for egocentric planning and task completion in embodied environments. They demonstrate their approach in ALFRED benchmark winning the 2022 CVPR challenge. Their central idea is the use of symbolic planners, which are typically used in fully observable settings. To deal with partial observability, they perform iterative exploration with symbolic planning. Structurally, their implementation is similar to the previous SOTA on ALFRED — FILM. However, instead of using a semantic map as FILM does, they use a graph structure for object and location information storage. This graph is updated through exploration and then used by the downstream symbolic planner.", "questions": "- Table1: main results should have bolded numbers and indications on which metrics need to be higher/lower for better task performance (e.g., with $\\uparrow$ and $\\downarrow$) to improve interpretability of results. Similarly sorting rows in Tab2 based on performance in unseen might be useful.\n- L166: “This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks” — not sure how?\n- Results: Given that the LGS-RPA also comes close in terms of GC and SR (unseen) to the authors approach, I encourage the authors to discuss LGS-RPA in the result section as well.\n- Tab:3 Unclear why the method only achieves ~60-70% GC with all ground truth available. Is this the upper bound? Why is the upper bound not 100% given that the env. is deterministic and we are using a symbolic planner?\n- Is the planning performed from scratch every time? I imagine that the state of the world and thus the graph have incremental changes so wondering if the authors do anything smart to reduce the planning time at each iteration.", "limitations": "- The authors do not touch upon the issue of obtaining PDDL domain in the first place, which is core to their approach. Unclear how this would scale for real world applications/agents and how errors in domain file can impact their approach. Lastly, they mention that they can handle additional constraints e.g., energy during planning. However, in practice, I’ve found planning time to blow up when using durative and cost constraints. Unclear again how authors approach would scale for more complex problems. Also, on that note, I’d like to see planner time numbers for the tasks perhaps in the supplementary. \n- Similarly, the authors say that they are more robust to perception errors however they say they do so using hand-crafted policies for replanning (Sec.8.3). Unclear if the robustness is because of these handcrafted policies or because of their iterative planning approach combining exploration and symbolic planning.\n- The  section on broader impact and societal impact is missing. I'd encourage the authors to think about real world applications that their work might enable for this section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors’ hybrid approach leveraging symbolic planners is novel amongst the modular approaches for long-horizon EAI tasks, and can be promising for future research.\n- The authors won the CVPR 2022 EAI ALFRED challenge.", "weaknesses": "- Poor clarity of exposition: I found the major aspects of authors’ technical approach poorly explained in the paper. This really limits reproducibility as well as use by the community IMO and consequently limits the value of the paper.\n    - Spatial graph: Unclear how the graph is built and what does it look like. The authors say that the graph encodes location as the node key and visual observations as values. However, it is unclear what is the co-ordinate system for location (consequently would we get the same graph for the same environment and task but different agent start position?), what happens when the agent visits the location more than once, and lastly, how is it used for the low-level policies that have to navigate given there is no map. \n    - Object-centric POMDP: The authors briefly mention the Cost POMDP but do not appropriately define or explain it at all in the main paper. Most definitions are pushed to supplementary. While it is okay to push nitty-gritty details to supplementary, without properly framing the problem and then grounding their approach in the framework, I am not sure if I can buy the authors’ claim that their approach is theoretically sound (L121).\n    - Exploration: Unclear how exploration is handled. The only description is in the intro (L52-56).\n    - The algorithm is not commented and contains symbols which are never defined or explained, making it difficult to understand what did the authors actually implement and why:\n        - Symbol Ge is never defined\n        - Unclear how cost c of action is computed and used in function $M^{PD}$.\n    - The authors say that the PDDL domain and environment definitions of actions are misaligned, which make sense (L149). However they never explain how this alignment is obtained in their approach. \n    - Furthermore, given the PDDL and planning terminologies might not be accessible/known to everyone, I encourage the authors to be more careful and clear in their descriptions. For instance, what exactly are anchor object types and why do we need them? Also unclear what other anchor object types can exist in EAI settings. \n    - I didn’t get much out of Sec.4 and 5 despite the fact that I work with symbolic planning and EAI and even when there is so much to explain about the approach as mentioned above. I recommend rewriting entire Sec.4 and 5.\n- Generalization: Authors claim and attempt to show generalization in Sec.8.4. However, this section had no details whatsoever about the new tasks they use nor a reference to supplementary. I managed to find some details in supplementary. However, I am sure a general reader will be lost here. Furthermore, the authors claim that they do better on the new tasks and achieve zero-shot success 82% times, better than other methods but do not show any such comparison in their work (tab.6 supplementary). Given this is one of the main claims of advantage of their approach, I’d like to see comparison with neural and template-based approachers, and also with the prompter method. \n\nWith the above (also see limitations section below), IMO the paper is not yet at par for publication at Neurips. In its current state, it can be a great workshop paper but requires major rewriting and additional comparisons e.g., for generalization claims otherwise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review.\nThere are indeed issues in the manuscript.\nWe apologize for the lack of clarity.\nAs we stated, we have already improved part of the issues in the manuscript and will certainly further improve it thanks to your feedback.\n\nThe general response addresses the following issues mentioned in this review:\n\n- Object-oriented POMDP and framing of the problem.\n- Spatial semantic graph.\n- Misalignment between the environment and the symbolic model.\n- The PDDL provided by the user.\n\nHere, we provide additional answers:\n\n> **Spatial graph**: See the attached PDF and the general response.\nThe coordinates are (x-coordinate, y-coordinate, facing direction).\nIf the agent visits the same location more than once and faces the same direction as before, that will not create a new node.\nVisiting the same room twice would create the same graph if we fixed the random seed for exploration, and all other components were deterministic.\nIf we let the agent explore the whole space in another order, the new resulting semantic graph would be equivalent to the previous one, except for object names created on the fly.\n\n> **Algorithm issue**: Apologies.\nThe algorithm has been fixed.\nFor instance, $G_e$ is now part of the Task that the algorithm receives as input.\nWe define Tasks in the shared response.\n\n> **Cost computation**: The generic algorithm supports a planner that minimizes or reduces cost.\nThe classical planner used in ALFRED does not use cost, but the plans tend to be short, as the domains are simple from the planning point of view.\n\n> **PDDL terminology**: We agree it might not be accessible.\nWe will improve the manuscript by providing timely examples and leave the formal definitions for the supplementary material.\n\n> **Generalization claim**: We fixed the references and improved the description to emphasize that we are solving new tasks, starting with a symbolic goal.\nSo, the 82% success rate is related to the 52.29% for w/ gt language in Table 3, valid unseen.\nThe comparison is not meaningful as these are different tasks.\n\n> **Comparisons**: Implementing the new tasks in Prompter and FILM++ (Inoue et al 2022), and FILM would require retraining the classifier with new tasks and creating templates by hand.\nWhile our results rely on creating a symbolic planning model, its compositionality allows us to request the agent with unforeseen tasks.\nThe best course of action depends on the domain.\n\n> **Publication readiness**: We hope this response has clarified how we will address the limitations of the current manuscript.\nPlease let us know if there is anything else that we could address, which might change your assessment regarding our submission’s acceptance.\n\n> **Table 1 and 2 recommendations**: Will do.\nFor all the metrics, higher is better.\nSuccess Rate demands to achieve the goal, while Goal-condition Success focuses on subgoals.\nHowever, in planning, achieving subgoals independently might not predict the success rate and have low value for the user.\n\n> **Clarification on robust action sequences**: We intended to say that our setup can solve tasks beyond the seven included in the ALFRED challenge.\nWe will improve Section 8.4 to make this connection clear.\nThe robustness of new tasks depends on the robustness of each action that might have been used in previous tasks.\n\n> **Anchor object types**: We discussed them in the general response.\nAn anchor type in an extension of ALFRED could be a panel that shows the power status of appliances.\n\n> **LGS-RPA comparison**: LGS-PRA is a very interesting piece of work.\nHowever, we believe that the improved accuracy is due to techniques on landmark detection method and local pose adjustment, which are not the emphasis of our work.\nWe find it difficult to conduct a proper comparison.\n\n> **Tab. 3 and upper bound**: There are two main reasons we cannot achieve 100% even with ground truth.\nFirst, the simulator can get stuck when holding large objects close to a wall, and some objects spawn in the corner cannot be reached due to fixed 90% turns.\nSecondly, our method does not build a graph for ‘turning up’ and ‘turning down’ actions, making some things unreachable.\n\n> **Planning process**: As classical planners expect a fixed set of objects, we plan from scratch every time we reach an exploration goal that might reveal new objects, or if there is an execution failure that we use for revising the semantic graph.\n\n> **PDDL scaling and constraints**: We discuss the obtention of PDDL in the general response.\nWriting, learning, or debugging PDDL leads to higher generalization, and the right decision depends on the domain.\nRegarding constraints, some can be mapped into classical planning with costs.\n\n> **Planner time numbers**: In the beginning, replanning is frequent but fast.\nWhen the graph is large enough, the plan can take up to 5 seconds but often takes around 1 second.\n\n> **Robustness**: Navigating around objects is a weak point of our approach.\nOur hand-crafted policy doubles down in this direction, following the ALFRED environment’s simplification.\nThe strength of our approach lies in exploiting skills and perception across tasks.\n\n> **Broader and societal impact**: Thank you.\nWe will elaborate on the potential benefits of our approach in this section, including robustness and higher explainability."}, {"review_id": "aIf2nua8AX", "reviewer": "Reviewer_dRsd", "summary": "This paper studies the problem of embodied tasks in which the agent needs to plan over long task horizons given natural language instruction. Current methods that use end-to-end training leads to entangled representation, which makes it hard to solve the task. On the other hand, planning methods such as PDDL can produce high-quality actions given a well-defined problem specification. To solve the task, the paper proposes a method that consists of two parts: (1) goal-oriented exploration that aims to gather missing information, and (2) a classical planning method that aims to produce a feasible action. The proposed method is evaluated on the embodied benchmark ALFRED. The method improves the prior SOTA performance by 8.3%, winning the ALFRED challenge at CVPR 2022 Embodied AI workshop.\n\nTo be more precise, the proposed method consists of several parts: (1) a visual module for semantic segmentation and depth estimation of the scene, (2) an egocentric planner for planning given the information gather, (3) a semantic spatial graph for scene memorization. At the beginning of the task, the agent is provided with 500 steps to explore its surroundings. After that, the information is converted to a semantic spatial graph for the input of PDDL. Figure 1 shows the method. The algorithm inerates using the following steps: (1) find a path for reaching the goal, (2) if the goal is not reached, do exploration again\n\nTable 1 shows the main result of the paper. And the rest of the experiments section provides a detailed ablation study of the method.", "questions": "1. If I want to deploy such a method in the real world, what would be the challenge and the additional step to do this?\n\n2. Following the previous questions, what is the Sim-to-real gap here?\n\n3. Will the method be able to handle tasks such as you need to open the drawer to find the coffee mug in it, instead of target objects being visible to the agent? I will say this is more challenging in the POMDP setting than the setting in the paper.\n\n4. What are some failure cases? Could you provide a couple of examples?\n\n5. Is it possible to integrate such a method with semantic exploration (https://devendrachaplot.github.io/projects/semantic-exploration)?", "limitations": "The paper has clearly stated the limitations of the paper, including the method cannot recover from irreversible failures, being sensitive to perception errors, and having no memory for belief tracking.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The writing of the paper is clear and easy to follow. For instance, in the introduction, I can easily understand the motivation of the method.\n\n2. The proposed method is a winning approach at the CVPR Embodied AI workshop in 2022. This shows that the method has been thoroughly tested, and the result is convincing.\n\n3. In the AI era, it is nice to see a classical planning method is robust in such tasks over a neural network-based method.\n\n4. The related work is reasonable. It covers important papers such as Saycan\n\n5. Overall, I think this is a good paper, and the method is elegant and promising.", "weaknesses": "The proposed method could be very specific to the ALFRED task, in the sense that the method is solely optimized and engineered for the  ALFRED task. For example, in some tasks, there is no abstraction of the action space, like OpenDrawer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. The connection with SayCan is interesting, as our work complements theirs.\n\n> **Weaknesses:**\n\n> The proposed method might be too specific to the ALFRED task, in the sense that the method is optimized solely for this task. For example, there's no abstraction of the action space, like OpenDrawer.\n\nWe agree that competitions—like metrics—often lead to specific optimizations. We tuned parameters, such as random exploration and object reachability depth. Yet, some optimizations are principled methods, like using graph reachability to eliminate irrelevant actions. This method is applicable beyond the ALFRED challenge.\n\n> If I want to deploy this method in the real world, what would be the challenges and additional steps?\n\nOur approach targets errors at the action level. For any application, ensure that actions perform as intended. In realistic environments, it is prudent to guarantee the robot's capabilities. Refer to our planning-based modular printer example.\n\n> What is the Sim-to-real gap here?\n\nRobust actions foster high accuracy. Ruml et al., JAIR 2011, describe a real-time modular printer. [Here's the video](https://www.jair.org/index.php/jair/article/view/10693). Noisy actions or unexpected results are outside our scope, but limited support is offered for failed actions.\n\n> Will the method handle tasks like opening a drawer to find a coffee mug?\n\nWhile we only support deterministic actions, classical planning can tackle partial observability. By following Albore, Palacios, Geffner, we can create hypothetical objects and optimistic actions. Upon interaction, if a mug is not found, the agent will seek it elsewhere.\n\n> What are some failure cases?\n\nFailures often arise from object recognition and distance estimation errors. An agent might misidentify an apple or collide with an obstacle. These failures are detailed in Table 2 on page 7.\n\n> Is it possible to integrate this method with semantic exploration?\n\nThank you for the reference. Our plans can play the role of the local policy $\\pi_L$ in SemExp, and we can integrate with other SLAM algorithms by using SLAM updates instead of our own. \n\n> **Limitations:**\n\n> The paper states the limitations, including irrecoverable failures, sensitivity to perception errors, and no memory for belief tracking.\n\nThank you. We do perform belief tracking in ALFRED. The semantic graph contains current information, and unvisited locations implicitly represent unknown objects and locations. Our method's open-world nature means we don't assume a finite number of objects. This relates to the 0-approximation notion by [Baral et al., AIJ 2000](https://www.sciencedirect.com/science/article/pii/S0004370200000436)."}, {"review_id": "1TW51lRCo0", "reviewer": "Reviewer_vi1s", "summary": "\nThe paper presents a modular approach that combines symbolic planning and object-centric Cost POMDP for solving ALFRED tasks. The proposed method demonstrates improvements compared to previous end-to-end and modular approaches, such as FiLM. Unlike methods like FILM, HLSM, and Prompter, EPA utilizes a semantic graph representation instead of a top-down 2D map.\n\nThe method incorporates an initial phase of 500 exploration steps to gather sufficient knowledge, which is crucial for determining an appropriate expandable initial state. Object information is selectively saved only when objects are within immediate reach (0.25m ahead). This selective saving facilitates the conversion of observations into a symbolic state and reduces the length of generated plans.\n\nThe authors' findings suggest that the performance enhancements achieved by EPA over FILM and similar approaches primarily stem from its iterative planning approach. This approach enables the agent to recover from failure scenarios through flexible subgoal ordering, leading to improved performance.\n\nAccording to the authors, EPA achieved the second-highest ranking on the ALFRED leaderboard, closely following the Prompter method. The superiority of Prompter, however, is attributed to modifications in obstacle size and reachable distance rather than the use of prompts.", "questions": "1. Could you kindly provide some clarification regarding the semantic spatial graph? Specifically, what information does it contain and what is the average size of such a graph? The description in Appendix F2 suggests that the graph incorporates visual observations, segmentation, and depth. I would like to understand if this graph represents the visual scene \"in front of\" the agent. If so, I have a couple of related questions:\n    1. In case the agent is at the same location but facing a different direction, would the information on the graph be overridden?\n    2. If not, does each node in the graph contain a representation of the \"360-degree visual observation\" at that particular location? How are the \"actions\" represented as edges in this context?\n2. I'm curious to know how the approach encodes common sense knowledge or visual knowledge (mentioned in Lines 57-58) as part of the domain definition in PDDL.\n3. Could you please explain how the \"exploration actions\" are defined? In Appendix F3, Listing 3, it is mentioned that the agent cannot hold something and explore the environment. For example, can the agent actively search for the coffee after picking up the cup? I would appreciate some clarification on this.\n4. I would like to understand how the possible predicates are listed in the PDDL. Additionally, how are the preconditions and effects identified for each predicate? Are they learned or inferred from visual observations, language goals, or the agent's interaction to gather information? Are these predicates hard-coded by a human? If so, I'm interested in understanding how this differs from the classical task planning setup.\n5. It is not entirely clear how the generalization to new tasks is achieved, particularly when the language module is trained to output a task type out of seven tasks (as mentioned in lines 73-74). Could you please elaborate on the meaning of the statement in lines 308-309, \"Our egocentric planning breaks a task into a set of goal states, autonomously generating an action sequence\"?", "limitations": "Overall, it appears that the current approach is heavily focused on reverse engineering the trajectory creation process for the ALFRED task, utilizing learned vision and language models. While the method demonstrates a high success rate on the ALFRED benchmark for unseen scenarios, there is room for improvement in terms of clarity and the overall significance of the proposed approach. It remains uncertain how applicable the use of semantic spatial graphs or PDDL domain definitions would be beyond the ALFRED simulated benchmark, particularly when considering real-world physical environments.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper demonstrates the use of preconditions and effects through PDDL to improve the overall success of long-horizon tasks, especially unseen success rate. \n2. The paper combines the use of symbolic planning using learned vision and language models and highlights how certain aspects of generalization can be achieved by abstraction.", "weaknesses": "1. The current assumptions on semantic spatial graphs require random exploration for 500 steps to visit each location and form a node in the graph. The paper reports a drop in performance if this initial observation phase is ignored. This approach has two major limitations: (1) it dramatically increases the timestep overhead for the proposed agent as compared to the existing works. (2) it assumes a static unchanging environment after mapping and will likely fail in realistic environments with dynamic obstacles. Given the existing visual-inertial SLAM approaches, as noted by the authors, it seems that this issue can be mitigated. Some existing approaches for topological mapping [1] might also be relevant, and in turn, improve the path length weighted success rate (PLWSR).\n\n1. Writing PDDL domain definitions and problems for each task is known to be a tedious coding task. Any errors in representing the available objects and actions would yield no plan. The current approach seems too close to reverse engineering the process of creating trajectories for the ALFRED task.  While the authors report 150 hours for PDDL domain and problem definitions, a large chunk of the work involving object types and action predicates is already described in the ALFRED metadata. This does not give a reasonable perspective on how many hours would it take to scale and maintain this approach further, especially in the physical world.\n\n *[1] Chaplot, D.S., Salakhutdinov, R., Gupta, A. and Gupta, S. 2020. Neural Topological SLAM for Visual Navigation. In CVPR.*\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Some of your comments were addressed in the general response.\n\n## Strengths\n\n> The paper demonstrates:\n> - The use of preconditions and effects through PDDL to improve the overall success of long-horizon tasks, especially unseen success rate.\n> - The combination of symbolic planning using learned vision and language models, highlighting how generalization can be achieved by abstraction.\n\nThose were part of our goals. Thank you.\n\n## Weaknesses\n\n> - The current assumptions on semantic spatial graphs require random exploration for 500 steps to visit each location and form a node in the graph.\n> - Writing PDDL domain definitions and problems for each task is tedious. While 150 hours were reported for PDDL domain and problem definitions, scaling and maintaining this approach's perspective is not clear, especially in the physical world.\n\nSee general response on PDDL construction and semantic spatial graphs.\n\n> [1] Chaplot, D.S., Salakhutdinov, R., Gupta, A. and Gupta, S. 2020. Neural Topological SLAM for Visual Navigation. In CVPR.\n\nThank you. We will add the citation. Very relevant.\n\n## Questions\n\n### Semantic Spatial Graph\n\n- **Information and Size:** The semantic map contains visual and depth information. It stores actionable items, their class, average pixel distance, obstacles, and more. The average size of such a graph is around 500 from our observation.\n- **Different Directions:** We encode unique nodes with different orientations, and there is no 360-degree observation. A detailed explanation of how the coordinates and edges are handled is provided.\n\n### Domain Definition in PDDL\n\n- **Common Sense and Visual Knowledge Encoding:** Extra goals can be added in PDDL, and LLM can help turn comments into extra PDDL goals.\n- **Predicates and Preconditions:** Planning domains are available, human-written, or learnable. Anchor types are identified, and an egocentric algorithm solves planning problems centered on the agent location.\n- **Generalization to New Tasks:** We tested on new tasks written directly as PDDL goals.\n\n### Exploration Actions\n\n- Exploration actions are defined to move to unexplored locations. They update the spatial graph and are affected by holding objects that obscure the view.\n\n### Generalization and Task Handling\n\n- **Language Module and Generalization:** Elaboration on how the egocentric planning breaks a task into goal states, autonomously generating an action sequence.\n- **Applicability and Reverse Engineering Concerns:** The approach focuses on ALFRED but generalizes to objects and planning actions beyond it. The reverse engineering concern and the point of ALFRED in generalization are addressed.\n\n## Limitations\n\n- **Clarity and Significance:** Improvement in terms of clarity and the overall significance of the proposed approach is needed.\n- **Real-world Applicability:** Uncertainty regarding the applicability of semantic spatial graphs or PDDL domain definitions beyond the ALFRED simulated benchmark, particularly in real-world physical environments.\n\nAnswer to physical \"environments\" will be provided in the general response."}, {"review_id": "vuMx4yr6sT", "reviewer": "Reviewer_JTSQ", "summary": "The authors propose an approach combining symbolic planning and object-oriented POMDPs for symbolic planning, which gets extremely strong performance on the ALFRED benchmark and won the CVPR ALFRED challenge. Their approach uses PDDL, but extends it with a set of exploration-focused actions. They use a combination of explicit knowledge and heuristics (exploring close to a seen object for example) that are learned from data. They explore for 500 steps, using this to build a spatial-semantic graph (instead of an explicit map like many previous methods). They then use this with an off-the-shelf task planner to choose which actions to execute.", "questions": "- How is the spatial graph constructed? It seems like a lot of the detail here is lacking.\n- How important are the 500 exploration steps? Seems like it should be able to work well without this, if it's really a good POMDP planner.\n- Construction of the initial state was also a bit unclear to me.", "limitations": "- Only applied to one benchmark\n- Need for 500 exploration steps at the beginning seems weird\n- Very dependent on perception models - perfect detection, depth, etc., which limits application to other domains.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Building a spatial semantic graph seems like a better (more scalable) way of solving ALFRED tasks than explicit maps\n- Strong performance on a well-respected benchmark\n- A lot of great ideas, and some of the explanation is very good\n- Great to see a *new* approach to solving ALFRED, not just building off of FILM/HLSM", "weaknesses": "- The name \"egocentric planning\" doesn't make a lot of sense to me; everything in ALFRED is going to be some manner of egocentric planning\n- Not clear how general this is - lots of engineering in the planning domain. More analysis would help here; ideally the authors could run on a different domain (OVMM might be one - ovmm.github.io), but there really aren't good options with strong existing baselines like ALFRED. Instead maybe they could describe in more detail how it would be applied to other domains. \n- Writing could be improved. Several typos (Apporach --> Approach), IER (exploration algorithm) not being referenced by name in text.\n- It's not really clear this is a *learning* contribution - learning components seem minor. I think this is ok, because it's still a useful result on a learning problem, but I could see the argument against it.\n- Very engineering heavy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We've addressed some of them in the general response.\n\n> **Strengths:**\n> \n> - A lot of great ideas; some explanations are very good\n> - Great to see a new approach to solving ALFRED, not just building off FILM/HLSM\n\nWe look forward to elaborating on these ideas.\n\n> **Weaknesses:**\n\n> - \"Egocentric planning\" doesn't make sense; everything in ALFRED is egocentric planning\n\nEgocentric planning emphasizes using a planner with full-observability for problems under partial observability. We intend to keep working in this direction. What would you suggest as a name?\n\n> - Unclear generality; lots of engineering in the planning domain\n\nSee the general response for a discussion on planning domains. Our method emphasizes scalability and could be a target for learning.\n\n> - More analysis needed; describe how it would apply to other domains\n\nOVMM is a possible domain, but physical manipulation is beyond our scope.\n\n> - Writing could be improved\n\nThank you. We'll correct typos and inconsistencies.\n\n> - Not clear this is a learning contribution\n\nOur focus is on integrating existing models. While perceptual models improve, multi-step generalization remains challenging.\n\n> - Very engineering heavy\n\nThe task required initial domain engineering to create a set of action object types. The planning actions should generalize to objects beyond ALFRED.\n\n- Generic: Perceptual models, Egocentric algorithm\n- Environment: actions, object types\n- Engineering: PDDL model, Semantic part updating\n- ALFRED tuning: exploration, policy changes\n\n> **Questions:**\n\n> - How is the spatial graph constructed?\n\nSee general response.\n\n> - Importance of 500 exploration steps?\n\nOurs is a lightweight POMDP solver. Early exploration is less useful. SLAM methods could be used to update semantic maps.\n\n> - Construction of the initial state?\n\nThe initial state comprises a location and perceived objects in the facing direction.\n\n> **Limitations:**\n\n> - Only applied to one benchmark\n\nIndeed, but our method might enable other applications with trained models for the AI2Thor environment.\n\n> - Very dependent on perception models\n\nOur contribution abstracts the perception model, widening applicability as generalist models improve.\n"}, {"review_id": "f2becuJMLp", "reviewer": "Reviewer_ZvXu", "summary": "The paper introduces Egocentric Planning, alternating exploration and task solving symbolic planning for long-horizon object-oriented POMDP in environments with deterministic action effects. The presented method is used as the planner in a hybrid agent, SOTA in the 2022 ALFRED benchmark, with neural perception. The agent is based on a previous successful design (FILM), with semantic SLAM in the latter replaced by a graph representing the current knowledge about the scene, besides the novel open loop planning algorithm exploiting this graph. The experimental results show a considerable improvement over FILM at the cost of longer trajectories required to succeed due to the time required to gather information about the scene, besides error analysis  and ablations. An exciting result is the generalization to new task types not present in training.", "questions": "1. How could the proposed framework be extended for multiple agents interacting with the environment?\n2. Related to one of the weaknesses, what is missing in the egocentric planner to be capable of successfully exploring the environment until enough information is gathered to solve the task?", "limitations": "I think the paper makes a good job describing limitations of the proposed method, but, as mentioned in Weaknesses, I think the judgment of better performing alternatives as baselines, given the acknowledged limitations of the proposed method, could be seen as unfair, so I think an improved contextualization is needed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method enables out-of-the-box generalization to new task types (within the same set of objects and relationships).\n- The graph representation and planner algorithm work as a tailored alternative to semantic SLAM for object-oriented POMDPs with remarkable performance.", "weaknesses": "- The presented approach is effective under the assumption of deterministic action effects, but in real-world usage this should not generally apply. The paper consequentially discusses some possible ways to overcome this limitation, but also claims some better performing alternatives should be considered as baselines rather than comprehensive solutions, since they are limited in scope. I'd encourage rephrasing that discussion, which as is could be interpreted as unfair to the alternatives.\n- The need for an exploration phase, when the egocentric planner is in principle designed to determine when to explore, seems a little ad hoc. I wish there was a clearer explanation why this cannot be provided by the main algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We appreciate the insight into our work in the multi-agent setting.\n\n> **Strengths:**\n> \n> The primary contribution is that our method enables generalized solutions for new task types, using the same set of objects and relationships.\n> \n> **Weaknesses:**\n> \n> Your point on the effectiveness under deterministic action effects is taken. Our method is distinct from those with native support for non-deterministic actions. While the egocentric algorithm doesn't require deterministic actions, our implementation for ALFRED does. This iterative algorithm is our baseline for studying high generalization, even in deterministic domains.\n> \n> The exploration phase, as you noted, seems ad hoc. We don't propose a universal solution but applied a specific one for ALFRED, enough to win the competition. This approach is open to integration with other SLAM methods, offering a promising avenue for improvement.\n> \n> **Questions:**\n> \n> - \"How could the proposed framework be extended for multiple agents?\"\n> \n>   A subset of multi-agent problems can be mapped into classical planning and adapted to our method, as per Muise et al. (2015).\n> \n> - \"What is missing in the egocentric planner to explore the environment until enough information is gathered?\"\n> \n>   The egocentric planner's current optimization restricts exploration due to its limitation on actions and the imperfection of a learned Unet. Our initial steps of exploration help generate reliable candidates, but with improved depth perception and budget, the current setup could explore effectively. Future work may weight the graph with rewards associated with locations where objects can be perceived, guiding A* to those locations.\n> \n> **Limitations:**\n> \n> We acknowledge your thoughts on the judgment of alternatives as baselines and will revise the manuscript. Our focus is on generalization, whereas other methods may have different priorities. \n"}]

Author rebuttal:
## Introduction

We would like to extend our heartfelt gratitude to the reviewers for their insightful comments and constructive criticism. The feedback has been encouraging and instrumental in helping us understand the significance of our work in the following areas:

- Our **modular approach** (R1, R3)
- The **role of symbolic planning** in solving a diverse set of tasks (R1, R2, R3)
- Our unique **notion of the semantic graph** (R4)
- The success we demonstrated in our entry in the **ALFRED competition** (all reviewers)

However, the reviewers have raised certain questions regarding the semantic graph, the symbolic planning model, and their relationship. We appreciate the opportunity to respond and provide clarity on these aspects.

## Core Contributions

Before diving into the detailed response, we wish to revisit the core of our contribution. Our focus is on environments with fixed, known types, relationships, and actions grounded on them. We aim to solve a series of tasks in the same environment, and our winning entry in the ALFRED challenge is a testament to the method that allows us to explain a particular implementation in depth.

### The Essence of Our Method

- We present a lightweight method for achieving goals that can be expressed and solved in the fixed environment.
- In ALFRED, we reuse existing perceptual models and employ a simple exploration strategy, so the novelty of our method resides in the planning part.
- While ALFRED features only seven classes of tasks, our method supports others as long as natural language processing can map the user goal into the fixed object types and relationships.
- We expect non-symbolic planning methods to perform better with more data or by focusing on specific tasks. However, our method serves as a simple baseline for studying systematic generalization and compositionality per domain.

### The ALFRED Challenge

- Our method's effectiveness is shown by our success in the ALFRED challenge.
- Though limited to seven task classes, our approach can support other tasks.
- We demonstrate additional tasks that can be solved in the implicit Object-Oriented POMDP, consisting of the ALFRED environment and standalone models for object detection and semantic parsing of the tasks.

### Object-Oriented POMDPs

R1 rightly pointed out issues in sections 4 and 5 of our manuscript describing Object-Oriented POMDPs and the core algorithm. In response:

- We have massively rewritten these sections.
- The definition of Object-Oriented POMDP is now more finely defined.
- In section 5, we define an Environment as a tuple \(\mathcal{E} = \langle A_\mathcal{E}, \mathcal{T_\mathcal{E}}, \mathcal{V_\mathcal{E}}, \mathsf{reset}, \mathsf{step} \rangle\), with various elements representing the set of parameterized actions, object types, initial observation, and action execution.
- Our method's goal is to solve a series of tasks in the same environment, which we define as a Task, \(T_\mathcal{E} = \langle I_\mathcal{E}, G_\mathcal{E} \rangle\), setting the initial state and goal of the environment.

### Observations and Planning

Our approach entails three major components:

1. **Object-Oriented Environment**: The environment encapsulates object types, potential values, and actions.
2. **Egocentric Planning Algorithm**: This algorithm uses deterministic actions and re-planning, focusing on a fragment of Object-Oriented POMDPs where the set of objects is initially unknown.
3. **Symbolic Planning Model**: Assumed to be provided by the user, the symbolic model may be non-trivial but can be argued to be necessary for generalization.

We also introduce the notion of **anchor objects**, which may reveal the existence of new objects. This concept generalizes to other domains, allowing the agent to discover new options while navigating different environments, such as a warehouse or a web application.

### Alignment between Symbolic Model and Environment

The alignment between the symbolic model and the underlying environment can be challenging. However, in ALFRED, this alignment is facilitated by a spatial semantic map. For each unique environment, we use a spatial graph to encode the agent's location and direction, with movements becoming the edges. This simplifies navigation and serves as a top-down map.

### Engineering Decisions

Some reviewers raised questions about our engineering decisions in ALFRED, like the exploration strategy. We will respond to these directly but emphasize that our ALFRED entry demonstrated our approach successfully.

## Additional Considerations

- **500 Steps**: We need to address the choice of limiting actions to 500 steps, explaining the reasoning and potential outcomes if this constraint is not used.
- **Sim2Real Limitations**: Our approach does not support continuous actions or complex manipulation. We assume discrete actions.
- **Perception and Multimodal Models**: We utilize multimodal models such as CLIP [14] and DETIC [26] for open-vocabulary object detection.
- **Potential Applications**: Our method could be applied in various scenarios, as described in table 1 of "HomeRobot: Open-Vocabulary Mobile Manipulation."

## Conclusion

Our response aims to shed light on our method's novelty, the alignment between the symbolic model and environment, and other specific questions raised by the reviewers. We hope to have provided a comprehensive overview of our work and look forward to further discussions and feedback. Thank you again for your valuable insights.


Meta-review: This paper leverages symbolic planners for effective egocentric planning over long-horizon task of following a instruction (in natural language). The authors demonstrate the efficacy of their simple baseline by achieving high performance on well-studied ALFRED benchmark in AI2THOR simulator. This authors do utilize an initial budget of 500 steps to explore/memorize the embodied agent's surroundings, which is represent as a semantic graph. The graph is transformed to a Planning Domain Definition Language (PDDL) formulation, making it amenable to classical planning approaches, which is done via an iterative exploration and replanning approach.

The reviewers are largely positive about the work and most of them concur that the paper has (1) a simple yet strong baseline for the standardized long-horizon planning benchmark of ALFRED, (2) effective use of symbolic planners, adding a novel take to end-to-end baselines in this research area, (3) good generalization, especially, in unseen scenes, and (4) good and clear presentation & writing. 

The concerns around (1) difficulty of obtaining a symbolic model, (2) restrictive empirical evidence, i.e., only on ALFRED, and (3) need for 500 steps as initial exploration are drawbacks that were also shared by the reviewers. Efforts towards stating them clearly and upfront will help. Adding ablations that relax the aforementioned assumptions, could help the community to fairly benchmark EPA (or the most appropriate variation of it) in future works.

The AC concurs with the overall positivity of the reviewers and their appreciation for this alternative take at tackling long-horizon interactive planning in embodied environments and effective generalization.
Decision: Accept (poster)