Title: Egocentric Planning for Scalable Embodied Task Achievement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Embodied AI

High-Level Actions

Symbolic Reasoning

Replanning

ALFRED Challenge

Flexible Task Achievement

User-Goal Understanding

Object Types and Actions

Perception Grounding
Abstract: Embodied agents face significant challenges when tasked with performing actions in diverse environments, particularly in generalizing across object types and executing suitable actions to accomplish tasks. Furthermore, agents should exhibit robustness, minimizing the execution of illegal actions. In this work, we present Egocentric Planning, an innovative approach that combines symbolic planning and Object-oriented POMDPs to solve tasks in complex environments, harnessing existing models for visual perception and natural language processing. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability, achieving an impressive 36.07\% unseen success rate in the ALFRED benchmark and winning the ALFRED challenge at CVPR Embodied AI workshop. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions, as well as what object types reveal information about others. It can naturally scale to solve new tasks beyond ALFRED, as long as they can be solved using the available skills. This work offers a solid baseline for studying end-to-end and hybrid methods that aim to generalize to new tasks, including recent approaches relying on LLMs, but often struggle to scale to long sequences of actions or produce robust plans for novel tasks.

Egocentric Planning for Scalable Embodied Task Achievement

Abstract

Embodied agents face significant challenges when tasked with performing actions in diverse environments, particularly in generalizing across object types and executing suitable actions to accomplish tasks. Furthermore, agents should exhibit robustness, minimizing the execution of illegal actions. In this work, we present Egocentric Planning, an innovative approach that combines symbolic planning and Object-oriented POMDPs to solve tasks in complex environments, harnessing existing models for visual perception and natural language processing. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability, achieving an impressive 36.07% unseen success rate in the ALFRED benchmark and winning the ALFRED challenge at CVPR Embodied AI workshop. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions, as well as what object types reveal information about others. It can naturally scale to solve new tasks beyond ALFRED, as long as they can be solved using the available skills. This work offers a solid baseline for studying end-to-end and hybrid methods that aim to generalize to new tasks, including recent approaches relying on LLMs, but often struggle to scale to long sequences of actions or produce robust plans for novel tasks.

1 Introduction

Embodied task accomplishment requires an agent to process multi-modal information and plan over long task horizons. Recent advancements in deep learning (DL) models have made grounding visual and natural language information faster and more reliable (MBP + 21). As a result, embodied task-oriented agents have been the subject of growing interest (STG + 20, YRT + 22, WDKM21). Benchmarks such as the Action Learning From Realistic Environments and Directives (ALFRED) were proposed to test embodied agents' ability to act in an unknown environment and follow language instructions or task descriptions (STG + 20). The success of DL has led researchers to attempt end-toend neural methods (ZC21, SGT + 21). In an environment like ALFRED, these methods are mostly framed as imitation learning, where neural networks are trained via expert trajectories. However, end-to-end optimization leads to entangled latent state representation where compositional and long-horizon tasks are challenging to solve. Other approaches use neural networks to ground visual information into persistent memory structures to store information (MCR + 22, BPF + 21). These approaches rely on templates of existing tasks, making them difficult to generalize to new problems or unexpected action outcomes. ALFRED agents must navigate long horizons and skill assembly, operating within a deterministic environment that only changes due to the agent's actions.

Christian Muise

Long sequential decision problems with sparse rewards are notoriously difficult to train for gradientbased reinforcement learning (RL) agents. But a symbolic planner with a well-defined domain can produce action sequences composed of hundreds of actions in less than a fraction of a second for many tasks. Embodied agents might need to conduct high-level reasoning in domains with long action sequences. In such scenarios, human experts can opt to model the task within a lifted planning language. Planning languages such as Planning Domain Definition Language , or PDDL (HLMM19), is the most natural candidate, given the high scalability of planners that use such languages (GB13). For instance, we define the tasks of a home cooking robot with actions, such as opening , move-to , pick-up , put-on , and objects, such as desk , stove , egg found in common recipes.

However, most symbolic planning techniques are relatively slow in partially observable environments with a rich set of objects and actions. Thus we propose an online iterative approach that allows our agent to switch between exploration and plan execution. We explicitly model the process of actively gathering missing information by defining a set of unknown-objects and exploration-actions that reveals relevant information about an agent's goal. Instead of planning for possible contingencies, we use an off-the-shelf classical planner(Hof01) for fully-observable environments to determine whether the current state has enough knowledge to achieve the goal. If more information is needed, our agent will switch to a goal-oriented exploration to gather missing information. Our approach can seamlessly incorporate any exploration heuristics, whether they are learned or specified.

We implemented and evaluated our method on the popular embodied benchmark ALFRED using only natural language task descriptions. To demonstrate the effectiveness of our method, we use the same neural networks for both visual and language grounding used in FILM, which was the SOTA and winner of the 2021 ALFRED challenge (MCR + 22). See a comparison with other methods in Table 1. By replacing FILM's template-based policies with our egocentric iterative planner, our method improved FILM's success rate (SR) by 8.3% (or a 30.0% relative performance increase) in unseen environments winning the the ALFRED challenge at CVPR 2022 Embodied AI workshop. Our empirical results show that the performance increase was attributed to a more robust set of policies that account for goal-oriented exploration and action failures. We also show that our method can conduct zero-shot generalization for new tasks using objects and actions defined in the ALFRED setting.

2 ALFRED Challenge

ALFRED is a recognized benchmark for Embodied Instruction Following (EIF), focused on training agents to complete household tasks using natural language descriptions and first-person visual input. Using a predetermined set of actions, the agent operates in a virtual simulator, AI2Thor (KMG + 17), to complete a user's task from one of seven specific classes instantiated in concrete objects. For example, a task description may be "Put a clean sponge on the table," and instructions that suggest steps to fulfill the task. The agent must conduct multiple steps, including navigation, interaction with objects, and manipulation of the environment. Agents must complete a task within 1000 steps with fewer than 10 action failures. We base our approach on task descriptions rather than the provided instructions. Task complexity varies, and an episode is deemed successful if all sub-tasks are accomplished within these constraints. Evaluation metrics are outlined in Section 8.

3 Overview of Egocentric Planning for ALFRED

Figure 1 illustrates our proposed method integrating several components to facilitate navigation and task completion: a visual module responsible for semantic segmentation and depth estimation, a language module for goal extraction, a semantic spatial graph for scene memorization, and an egocentric planner for planning and inference. Initially, high-level language task description is utilized to extract goal information, and the agent is provided with a random exploration budget of 500 steps to explore its surroundings. The random exploration step is used to generate diverse set of object cluster for further exploration using our planner. Subsequently, at t = 500, the gathered information from the semantic spatial graph is converted into a PDDL problem for the agent. We employ a novel open-loop replanning approach, powered by an off-the-shelf planner, to support further exploration and goal planning effectively.

DirectedGraph

NL Instruction

Nodes:

Put a clock on the

(xyo):

Table

[ObjMask,Depth,

Class, ...

Edges:

Action: Left,Right,

Segmentation

Forward

Goal:

(On Clock Table)

InitialCondition:

(OnBreadTable)

(TomatoSliced)

(agent at f0-3-5f)

Depth

Spatial Semantic

SensoryInputs

VisionModule

Graph

EgocentricPlanner

Exp Action

a_exp.

Plan Action

Environment

a_plan,

To facilitate egocentric planning, we first define the ontology of the planning problems, which encompasses action schemas, object types, and potential facts schemas. The natural language task description is subsequently converted into a planning goal. Following the initial exploration phase, the semantic spatial graph is updated to indicate the new position of the agent and its perceptions. The algorithm iterates through these steps: a) finding a plan for achieving the goal and returning it upon success; b) if no plan exists, replacing the goal with another fact, called (explore) , associated with unvisited states. The semantic spatial graph is updated during each iteration to construct the new initial state of the agent, allowing for an incremental egocentric view of the environment.

4 Embodied Agents: Generalization within Object Types and Skills

Embodied Agents are task solvers in environments linked to a hidden Object-oriented POMDP, emphasizing objects, actions/skills, and costs rather than rewards. We assume that these objects mediate perceptions and actions. An Environment is a tuple, E = ⟨ A E , T E , V E , reset , step ⟩ , with A E being the set of parameterized actions, T E defining the set of object types and potential values V E , reset representing the first observation, and step representing an action execution that returns an observation and cost. A Task is represented by a tuple, T E = ⟨ I E , G E ⟩ , which sets the initial state and the goal of the environment via the reset function. Lastly, Embodied Agents employ functions M I to initialize their mental state and M U to update it, have a policy π M to select actions and a function G M for deciding when they consider the goal to be achieved.

Agents that tackle a stream of tasks within the same environment have incentives to specialize to the known object types and their inherent capabilities expressed through parameterized actions. Such over-specialization provides an opportunity for developing robust agents that can generalize across different tasks within the same environment. This principle applies to physical and software agents whose actions are grounded in these objects. Our framework forms a basis for creating flexible and adaptable agents that can perceive and act upon recognized object types. Further details and implications are covered in Section D.

5 Egocentric Planning for Embodied Agents

Here, we introduce Egocentric Planning , an approach for embodied agents to solve tasks in Objectoriented POMDP environments. The approach requires a symbolic planning model that assumes full observability for testing both action applicability and goal achievement. However, the complexity of specifying or learning such a symbolic model is simplified by its alignment with the Object-oriented POMDP environment, object types, and action signatures. Egocentric planning relies on a reasonable assumption to derive a sensor model, equivalent to a function that updates the distribution of possible

current states given new observations (APG09). For instance, in a planning domain with objects for different types and colors, the sensor model can say that cubes and rectangles are red, while ovals and circles are black. When the agent examines an object, and observes that it is black, the agent should filter the belief to represent that the object cannot be a cube or a rectangle. In contrast, our method requires a set of anchor object types that are assumed to reveal information about other objects and exploration actions that allow visiting new anchor objects, revealing new objects and their properties. The method can be combined with standalone models for processing images or natural language. The resulting method is both theoretically sound, assuming that the symbolic planning model is correct, and practically applicable, as it leverages the speed of symbolic planners.

We present an abstract algorithm that leverages the objects types and action signatures of the Object-oriented POMDP to ground the required symbolic planning model.

5.1 Background: Parameterized Full-Observable Symbolic Planning

In the supplementary material, Section E, we define Parameterized Full-Observable Symbolic Planning domains and problems . A planning domain is defined as a tuple PD = ⟨T , V , P , A , Φ ⟩ where T is a finite set of object types with V a finite set of values, P is a finite set of typed predicates, A is a finite set of typed actions with preconditions and effects, and Φ is a function that maps each action and predicates to a tuple expressing their typed arguments. A planning problem consists of a planning domain, a finite set of objects, and descriptions of the initial state and the goal.

For ALFRED, we rely on Classical Planning , a kind of full-observable planning that assumes deterministic actions, where preconditions are a set of predicates, and effects are a pair of sets of predicates called add and delete, representing the action's modifications. Correspondingly, a parametric classical planning problem is a full-observable planning problem where the initial state and the goal are conjunctions of grounded predicates.

5.2 Egocentric Planning for Embodied Agents

Egocentric planning employs a user-specified fully-observable symbolic planning model to address tasks in Object-oriented POMDP environments. In the next section, we detail the specifics of our ALFRED implementation. As actions in ALFRED are deterministic, we can apply a classical planner to solve the associated planning problems. We first present a general algorithm for parametric full-observable planning problems.

Algorithm 1 implements our Egocentric Planning approach by constructing a sequence of planning problems, alternating exploration and task solving. At each step, the agent's mental state is updated though sensory input to represent the current state and the known objects that will be part of the

following planning problem. The agent deems the goal achieved when it finds a plan for achieving it and executes the plan successfully. Our policy ( π M ) leverages a standalone planner for both exploration and task solving.

Our exploration and sensing strategy hinges on two elements. First, anchor object types T a ⊆ T are types of objects that reveal the presence of others. For example, in ALFRED, location is the sole anchor object type. Second, exploration actions X ⊆ A enables the discovery of previously unknown objects. For instance, moving to a new location in ALFRED enables observing new items and locations. This object-centric view of the environment simplifies the creation of a symbolic model required for planning. The function ExploreAct ( a , o ) returns a copy of the function where (unknown o ) is added to the precondition, and the effect is extended to cause (explored) to be true and (unknown o )) to be false.

Our agent's mental state is represented by a set of objects and an initial state of grounded predicates, ( O , I ) . The function M PD I initializes the symbolic goal G and sets ( O , I ) with the initial objects and true grounded predicates. The mental state update function, M PD U , updates ( O , I ) given the executed action a , the returned perception p , and the cost c of executing a . For example, the mental state update in ALFRED adds new objects observed in the perception. It updates the initial state with these new objects and their properties, including their relationship with non-perceived objects like location. Both functions maintain the alignment between the current planning state and the environment with an underlying object-oriented POMDP model.

Specifically, the actions, object types, and the values of the environment may be different from the planning domain's. While the ALFRED environment supports movement actions like move-forward or turn-left , the planning domain might abstract these into a move action. Although the ALFRED environment does not include a location type, we manually specify it so movable objects can be associated with a specific location. Indeed, the first initial state I includes a first location and its surrounding ones. Thus, the agent can revisit the location once it has found a solution involving that object. While integrating these components may be challenging, it is a hurdle common to all embodied agent approaches aiming to generalize across components under a constrained training budget. In general, practical applications will likely monitor the agent's performance regarding object types and skills. We discuss the advantages and drawbacks of our approach in Section 9.

6 Approach for ALFRED

6.1 Model Overview

Our method is structurally similar to FILM but we substitutes the SLAM-style map with a graph structure for object and location information storage (MCR + 22). The action controller employs our iterative egocentric planner that utilizes a semantic location graph for task completion and exploration. The node of the graph are uniquely labeled by the coordinates (x-coordinate, y-coordinate, facing direction), and the edges are labeled by actions(turn-right, turn-left, move-forward). Each node in the graph stores object classes, segmentation masks, and depth information generated by the vision module. We label the nodes with "known" if the agent has visited the particular location and "unknown" if the node is only observed. Unknown nodes on the graph are prioritized for exploration based on objects they contain. This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks. At each timestep, the vision module processes an egocentric image of the environment into a depth map using a U-Net, and object masks using Mask2-RCNN (HGDG17, LCQ + 18). The module then computes the average depth of each object and stores only those with an average reachable distance less than 1.5. A confidence score, calculated using the sum of the object masks, assists the egocentric planner in prioritizing object interaction. We employ text classifier to transform high-level language task description into goal conditions. Two separated models determine the task type and the objects and their properties. For instance, a task description like "put an apple on the kitchen table" would result in the identification of a "pick-up-and-place" task and a goal condition of (on apple table) . We convert the FILM-provided template-based result into goal conditions suitable for our planner. The spatial graph, acting as the agent's memory during exploration, bridges grounded objects and the state representation required for our planner. The graph, updated and expanded by policies produced by our egocentric planner, encodes location as the node key and visual observations as values, with edges representing agent actions.

6.2 Egocentric Agent for ALFRED

The object-oriented environment for ALFRED, denoted as E = ⟨ A E , T E , V E , reset , step ⟩ , includes actions such as pick-up, put-down, toggle, slice, move-forward, turn-left, and turn-right. The FILM vision module is trained as flat recognizer of all possible object types. We postprocess such detection into high level types Object and Receptacle in T E , with corresponding V E indicating subtypes and additional properties. For instance, the object type includes subtypes like apple and bread, while Receptacle includes subtypes such as counter-top and microwave. Values incorporate properties like canHeat, canSlice, isHeated, and isCooled.

The planning domain for ALFRED is denoted PD = ⟨T , V , P , A⟩ . T includes the Location type, with corresponding values in V representing inferred locations as actions are deterministic. Predicates in P represent the connection between locations and the presence of objects at those locations. Further predicates express object properties, such as canHeat, canSlice, isHeated, and isCooled. Unlike the ALFRED environment, actions in the planning domain are parametric and operate on objects and locations. Instead of move-forward, the planning domain features a move action, representing an appropriate combination of move-forward, turn-left, and turn-right. For actions like toggle in ALFRED, which have different effects depending on the object in focus, we introduce high-level, parametric actions like heat that can be instantiated with the relevant object. As the actions in ALFRED are deterministic, we use a classical planner, enabling high scalability by solving one or two fresh planning problem per iteration of Algorithm 1.

7 Related Work

Visual Language Navigation (VLN) involves navigating in unknown environments using language input and visual feedback. In this context, we focus on the ALFRED dataset and methods based on the AI2-Thor simulator, which serve as the foundation for various approaches to embodied agent tasks. Initial attempts on ALFRED employed end-to-end methods, such as a Seq2Seq model (STG + 20), and transformers (ZC21). However, due to the long episodic sequences and limited training data, most top-performing VLN models for ALFRED employ modular setups, incorporating vision and language modules coupled with a higher-level decision-making module (MCR + 22, MC22, IO22). These methods typically rely on hand-crafted scripts with a predefined policy. However, this approach has several downsides. First, domain experts must manually specify policies for each new task type, regardless of the changes involved. Second, fixed-policy approaches do not allow the agent to recover from errors in the vision and language modules, as the predefined policy remains unchanged.

Recently, there has been growing interest in utilizing planning as a high-level reasoning method for embodied agents. Notable work in this area includes OGAMUS (LSS + 22), and DANLI (ZYP + 22). These methods demonstrate the effectiveness of planning in achieving more transparent decisionmaking compared to end-to-end methods while requiring less domain engineering than handcrafted approaches. However, these works primarily tested in simpler tasks or rely on handcrafted planners specifically tailored to address a subset of embodied agent problems.

Large Language Models (LLMs) have recently been harnessed to guide embodied actions, offering new problem-solving approaches. Works like Ichter et al.'s SayCan utilize LLMs' likelihoods and robots' affordances, grounding instructions within physical limitations and environmental context (iBC + 23). Song et al.'s (SWW + 23) LLM-Planner and Yao et al.'s (YZY + 23) ReAct have further demonstrated the potential of LLMs for planning and reasoning for embodied agents. Alternative methods not using LLMs, include Jia et al. and Bhambri et al.'s hierarchical methods utilizing task instructions(JLZ + 22, BKMC22). However, the scalability and robustness of these methods is under study. Our approach, in contrast, is ready to capitalize on the expressivity of objects and actions for superior generalization in new tasks, ideal for rich environments with multiple domain constraints. This challenges of using LLM for symbolic planning is noted by Valmeekam et al. (VSM + 23) who showed the weakness of prompting-based approaches, and Pallagani et al. (PMM + 22), who highlight the massive amount of data required for reaching high accuracy.

8 Experiments and Results

The ALFRED dataset contains a validation dataset which is split into 820 Validation Seen episodes and 821 Validation Unseen episodes. The difference between Seen and Unseen is whether the room

environment is available in the training set. We use the validation set for the purpose of fine-tuning. The test dataset contains 1533 Test Seen episodes and 1529 Test Unseen episodes. The labels for the test dataset are contained in an online server and are hidden from the users. Four different metrics are used when evaluating ALFRED results. Success Rate (SR) is used to determine whether the goal is achieved for a particular task. Goal-condition Success is used to evaluate the percentage of subgoal-conditions met during an episode. For example, if an objective is to "heat an apple and put it on the counter," then the list of subgoals will include "apple is heated" and "apple on counter." The other two metrics are Path Length Weighted by Success Rate (PLWSR) and Path Length Weighted by Goal Completion (PLWGC), which are SR and GC divided by the length of the episode. For all the metrics, higher is better. We compare our methods, Egocentric Planning Agent (EPA), with other top performers on the current ALFRED leaderboard. Seq2Seq and ET are neural methods that uses end-to-end training (STG + 20, ZC21). HLSM, FILM, LGS-PRA are hybrid approach that disentangles grounding and modeling (BPF + 21, MCR + 22, MC22, IO22).

8.1 Results

Table 1 presents a comparison of our outcomes with other top performing techniques on the ALFED dataset. With an unseen test set success rate of 36.07% and a seen test set rate of 44.14%, our EPA method won the last edition of the ALFRED challenge. The ALFRED leaderboard ranks based on SR on unseen data, reflecting the agent's capacity to adapt to unexplored environments. At the moment of writing, our achievements rank us second on the ALFRED leaderboard. The only method surpassing ours is Prompter(IO22), which utilizes the same structure as FILM and incorporates search heuristics-based prompting queries on an extensive language model. However, their own findings indicate that the performance enhancements are largely attributed to increasing obstacle size and reducing the reachable distance represented on the 2D map. The other top performer is LGS-PRA, which is also based on the FILM architecture, that improves performance via techniques on landmark detection method and local pose adjustment. In contrast, EPA employs a semantic graph, a representation distinct from the top-down 2D map applied by FILM, HLSM, LGS-PRA and Prompter. Our findings suggest that the performance boost over FILM and similar methods stems from our iterative planning approach, which facilitates recovery from failure scenarios via flexible subgoal ordering. Techniques outlined in LGS-PRA and Prompter should be compatible with our approach as well. The integration of these techniques with EPA an interesting area to investigate for future works. Our method exhibits lower PLWSR and PLWGC due to our initial 500 exploration steps. These steps are convenient for our planner to amass a diverse of of objects clusters and action angles to determine a proper expandable initial state. Although a pure planning-based exploration would eventually provide us with all the information required to solve the current task, it tends to excessively exploit known object clusters before exploring unknown areas. We only save object information when they are within immediate reach which means the agent is acting on immediate observations. This is done for convenience of converting observation into symbolic state and to reduce the plan length generated by the planner. Additional integration with a top-down SLAM map should help decrease the initial exploration steps, consequently enhancing both PWSR and PLWGC.

Table 2: Percentage Error of each types in the validation set.

Table 3: Ablation study with ground truth(w/ gt) of different perceptions

8.2 Ground Truth Ablation

Our model, trained on the ALFRED dataset, incorporates neural perception modules to understand potential bottlenecks in visual perception and language, as explored via ablation studies detailed in Table 3. We observed modest enhancements when providing the model with ground truth depth and segmentation, with ground truth language offering the most significant improvement (18.22% and 19.87%). Our egocentric planner, responsible for generating dynamic policies through observation and interaction, proves more resilient to interaction errors compared to policy-template based methods like HLSM and FILM (MCR + 22, BPF + 21). Thus, having the correct interpretation of the task description impacts our method more significantly than ground truth perception. We examined common reasons for failed episodes using both the seen and unseen datasets. The most prevalent error (23.54% unseen) is the inability of the agent to locate objects of interest due to failed segmentation or an unreachable angle. Errors stemming from language processing, where the predicted task deviates from the ground truth, are given priority. The results can be seen in Table 2.

8.3 Parameter Ablation

Neural vision modules may struggle with out-of-distribution data, such as ALFRED's unseen environments, leading to potential errors in object segmentation and distance estimation. Unlike hybrid methods like FILM and HLSM, our goal-oriented egocentric planner reassesses the environment after failure, generating new action sequences. For instance, in a task to hold a book under a lamp, a failure due to obscured vision can be addressed by altering the action sequence, as illustrated by the drop in performance to 27.31% in our ablation study (Table 4).

FILM's semantic search policy, trained to predict object locations using top-down SLAM maps, differs from our symbolic semantic graph representation, which precludes us from adopting FILM's approach. Instead, we hand-crafted two strategies to search areas around objects and receptacles based on common sense knowledge, which improved our Success Rate by 7.22% with no training. However, our agent only forms a node when it visits a location, necessitating an initial observation phase. Neglecting this step resulted in a 9.35% performance drop. FILM's SLAM-based approach can observe distant objects, requiring fewer initial observation steps and leading to better PLWSR and PLWGC. Combining a top-down SLAM map with our semantic graph could mitigate the need for our initial observation step and improve these scores.

Our algorithm's frequently re-plans according to new observation which is also essential for realworld applications using online planning system. Using modern domain-independent planners, such

as Fast Downward and Fast Forward (Hof01, Hel06), we've significantly reduced planning time. We tested a blind breadth-first search algorithm but found it impractical due to the time taken. A 10-minute cut-off was introduced for each task, which didn't impact our results due to an average execution time of under 2 minutes. Yet, this constraint severely lowered the success rate of the blind search method, only completing 3.47% of the validation unseen tasks (Table 4).

8.4 Generalization to Other Tasks

Our approach allows zero-shot generalization to new tasks within the same set of objects and relationships, offering adaptability to new instructions without manually specify new task types and templates. While neural network transfer learning remains challenging in embodied agent tasks, and FILM's template system requires hand-crafted policies for new tasks, our egocentric planning breaks a task into a set of goal conditions, autonomously generating an action sequence. This allows our agent to execute tasks without transfer learning or new task templates. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made to minimize perception errors. We tested this on five new task types not in the training data, achieving an 82% success rate, as shown in Table 6, sectionC.2 of the Supplementary Material. We could also adapt to new objects, actions, and constraints, such as time and resource restrictions. However, the current ALFRED simulator limits the introduction of new objects and actions. This flexibility to accommodate new goals is a key advantage of our planning-based approach.

9 Discussion

Our contribution poses Embodied Agents as acting in a Partially Observable Markov Decision Process (POMDP), which allows us to tackle a broad class of problems, and further expands on the potential applications of planning-based methods for embodied agents. By adopting this approach, our method offers a more flexible and adaptable solution to address the challenges and limitations associated with fixed-policy and handcrafted methods.

We found that using intentional models with a factored representation in embodied agents is crucial. Whereas a learned world model provides a degree of useful information, their associated policies might under-perform for long horizon high-level decision-making, unless they are trained on massive amounts of interactions (SSS + 17). Comparing a world model with classical planning uncovers interesting distinctions. While creating a PDDL-based solution requires an initial investment of time, that time is amortized in contexts that require quality control per type and skill. On the other hand, the cost spent on crowdsourcing or interacting with a physical environment can be significant (STG + 20).

Learning classical planning models is relatively straightforward with given labels for objects and actions (AFP + 18, CDVA + 22). On the other hand, if we aim to learn a world model, we should consider the performance at planning time and end-to-end solutions may not be the ideal approach in this context. When compared to intentional symbolic-based methods, end-to-end strategies tend to compound errors in a more obscure manner. Furthermore, as learned world models rely on blind search or MCTS, in new tasks the model might waste search time in a subset of actions that are irrelevant if action space is large, while symbolic planning can obtain long plans in a few seconds. We also propose addressing challenging planning tasks using simpler planning techniques, providing a foundation for more complex planning paradigms. Our method could be further enhanced by encoding explicit constraints and resources, such as energy.

While other methods might outperform our model in the ALFRED benchmark, they are often limited in scope. For instance, despite its high unseen success rate, the Prompter model serves as a baseline rather than a comprehensive solution for embodied agents. Our contribution provides a method that can adapt to shifts in task distribution without changing the symbolic planning model, while also supports changes in the agent's capabilities by modifying the symbolic planning model.

We acknowledge certain limitations in our approach. While we support safe exploration assuming reversible actions, real scenarios beyond ALFRED could include irreversible actions and failures. Our model is also sensitive to perception errors, making end-to-end methods potentially more robust in specific scenarios. Furthermore, our perception model lacks memory, potentially hindering nuanced

belief tracking. Despite these challenges, our method offers a promising direction for future research in embodied agent applications. In section B, we expand on the discussion of our approach.

10 Conclusion and Future Work

Our work introduces a novel iterative replanning approach that excels in embodied task solving, setting a new baseline for ALFRED. Using off-the-shelf automated planners, we improve task decomposition and fault recovery, offering better adaptability and performance than fixed instruction and end-to-end methods. Moving forward, we aim to refine our location mapping for precise environment understanding, extract more relational data from instructions, and integrate LLMs as exploration heuristics. We plan to tackle NLP ambiguity in multi-goal scenarios by prioritizing and pruning goals based on new information. Moreover, we aim to upgrade our egocentric planner to manage non-deterministic effects, broadening our method's scope beyond ALFRED. This direction, leveraging efficient planning methods for non-deterministic actions, is promising for future research (MBM14).

Acknowledgments

We want to acknowledge the ServiceNow research for providing the research opportunity and computational infrastructure to conduct this research project.This work was also made possible due to the assistance of Cyril Ibrahim at ServiceNow during the experimental execution phases of our project. We would also like to thank members of Mulab at Queen's University for providing a supporting research environment. Lastly, we want to acknowledge the assistance from organizers of the CVPR Embodied Agent Workshop for their guidance and feedback throughout the competition(DBB + 22).

References

[AFP + 18] Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review , 33(e20):1-25, 2018.

[AKFM22] Masataro Asai, Hiroshi Kajino, Alex Fukunaga, and Christian Muise. Classical planning in deep latent space. J. Artif. Intell. Res. , 74:1599-1686, 2022.

[ALPK18] Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. In Conference on robot learning , pages 856-868. PMLR, 2018.

[APG09] Alexandre Albore, Héctor Palacios, and Hector Geffner. A translation-based approach to contingent planning. In Twenty-First International Joint Conference on Artificial Intelligence , 2009.

[BKMC22] Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Hierarchical modular framework for long horizon instruction following, 2022.

[BPF + 21] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRL , volume 164 of Proceedings of Machine Learning Research , pages 706-717. PMLR, 2021.

[CC04] Stephen Cresswell and Alexandra Coddington. Compilation of ltl goal formulas into pddl. In ECAI , volume 16, page 985, 2004.

[CDVA + 22] Ethan Callanan, Rebecca De Venezia, Victoria Armstrong, Alison Paredes, Tathagata Chakraborti, and Christian Muise. Macq: A holistic view of model acquisition techniques. In ICAPS 2022 Workshop on Knowledge Engineering for Planning and Scheduling , 2022.

[CMW13] Stephen N. Cresswell, Thomas L. McCluskey, and Margaret M. West. Acquiring planning domain models using LOCM. The Knowledge Engineering Review , 28(2):195213, 2013.

[CST + 22] Rohan Chitnis, Tom Silver, Joshua B Tenenbaum, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Learning neuro-symbolic relational transition models for bilevel planning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4166-4173. IEEE, 2022.

[DBB + 22] Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, and Jiajun Wu. Retrospectives on the embodied ai workshop, 2022.

[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pretraining of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019.

[DNR + 20] Srijita Das, Sriraam Natarajan, Kaushik Roy, Ronald Parr, and Kristian Kersting. Fitted q-learning for relational domains. arXiv preprint arXiv:2006.05595 , 2020.

[DV13] Guiseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. In Proc. IJCAI 2013 , pages 854-860, 2013.

[FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

[GA14] Ilche Georgievski and Marco Aiello. An overview of hierarchical task network planning. arXiv preprint arXiv:1403.7426 , 2014.

[GB13] Hector Geffner and Blai Bonet. A concise introduction to models and methods for automated planning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 8(1):1-141, 2013.

[GNT04] Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated planning - theory and practice . Elsevier, 2004.

[HBB + 19] Daniel Höller, Gregor Behnke, Pascal Bercher, Susanne Biundo, Humbert Fiorino, Damien Pellier, and Ron Alford. HDDL - A language to describe hierarchical planning problems. In Second ICAPS Workshop on Hierarchical Planning , 2019.

[Hel06] Malte Helmert. The fast downward planning system. J. Artif. Intell. Res. , 26:191-246, 2006.

[HGDG17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV , pages 2980-2988. IEEE Computer Society, 2017.

[HLMM19] Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, and Christian Muise. An Introduction to the Planning Domain Definition Language . Morgan & Claypool, 2019.

[Hof01] Jörg Hoffmann. FF: the fast-forward planning system. AI Mag. , 22(3):57-62, 2001.

[HPBL23] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

[iBC + 23] brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, KuangHuei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as i can, not as i say: Grounding language in robotic affordances. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research , pages 287-318. PMLR, 14-18 Dec 2023.

[IO22] Yuki Inoue and Hiroki Ohashi. Prompter: Utilizing large language model prompting for a data efficient embodied instruction following. arXiv preprint arXiv:2211.03267 , 2022.

[JLZ + 22] Zhiwei Jia, Kaixiang Lin, Yizhou Zhao, Qiaozi Gao, Govind Thattai, and Gaurav S. Sukhatme. Learning to act with affordance-aware multimodal neural SLAM. In International Conference on Learning Representations , 2022.

[KHW95] Nicholas Kushmerick, Steve Hanks, and Daniel S. Weld. An algorithm for probabilistic planning. Artif. Intell. , 76(1-2):239-286, 1995.

[KMG + 17] Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: an interactive 3d environment for visual AI. CoRR , abs/1712.05474, 2017.

[KSV + 22] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 12262-12267, 2022.

[LCQ + 18] Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from CT volumes. IEEE Trans. Medical Imaging , 37(12):2663-2674, 2018.

[LSS + 22] Leonardo Lamanna, Luciano Serafini, Alessandro Saetti, Alfonso Gerevini, and Paolo Traverso. Online Grounding of Symbolic Planning Domains in Unknown Environments. In Proceedings of the 19th International Conference on Principles of Knowledge Representation and Reasoning , pages 511-521, 8 2022.

[MBM14] Christian J. Muise, Vaishak Belle, and Sheila A. McIlraith. Computing contingent plans via fully observable non-deterministic planning. In AAAI , pages 2322-2329. AAAI Press, 2014.

[MBP + 21] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

[MC22] Michael Murray and Maya Cakmak. Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment. IEEE Robotics Autom. Lett. , 7(3):6870-6877, 2022.

[MCR + 22] So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. FILM: following instructions in language with modular methods. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

[MCRW09] TLMcCluskey, SN Cresswell, NE Richardson, and MM West. Automated acquisition of action knowledge. In 1st International Conference on Agents and Artificial Intelligence , pages 93-100. SciTePress, 2009.

[MH13] Christoph Mies and Joachim Hertzberg. Encoding htn heuristics in pddl planning instances. In KI 2013: Advances in Artificial Intelligence: 36th Annual German Conference on AI, Koblenz, Germany, September 16-20, 2013. Proceedings 36 , pages 292-295. Springer, 2013.

[MK12] Mausam and Andrey Kolobov. Planning with Markov Decision Processes: An AI Perspective . Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2012.

[MMB14] Christian J. Muise, Sheila A. McIlraith, and Vaishak Belle. Non-deterministic planning with conditional effects. In Proc. ICAPS 2014 , pages 370-374, 2014.

[MvSE22] Jorge A Mendez, Harm van Seijen, and ERIC EATON. Modular lifelong reinforcement learning via neural composition. In International Conference on Learning Representations , 2022.

[NAI + 03] Dana S. Nau, Tsz-Chiu Au, Okhtay Ilghami, Ugur Kuter, J. William Murdock, Dan Wu, and Fusun Yaman. SHOP2: An HTN planning system. Journal of Artificial Intelligence Research , 20:379-404, 2003.

[ORCC21] Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems , 34:30298-30312, 2021.

[PG09] Hector Palacios and Hector Geffner. Compiling uncertainty away in conformant planning problems with bounded width. Journal of Artificial Intelligence Research , 35:623-675, 2009.

[PMM + 22] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv preprint arXiv:2212.08681 , 2022.

[SAH + 20] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

[SB09] Scott Sanner and Craig Boutilier. Practical solution techniques for first-order mdps. Artificial Intelligence , 173(5-6):748-788, 2009.

[SGT + 21] Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav S. Sukhatme. Embodied BERT: A transformer model for embodied, language-guided visual task completion. CoRR , abs/2108.04927, 2021.

[SSS + 17] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550(7676):354-359, 2017.

[STG + 20] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR , pages 10737-10746. Computer Vision Foundation / IEEE, 2020.

[SWW + 23] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models, 2023.

[TGD04] Prasad Tadepalli, Robert Givan, and Kurt Driessens. Relational reinforcement learning: An overview. In Proceedings of the ICML-2004 workshop on relational reinforcement learning , pages 1-9, 2004.

[VSM + 23] Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambhampati. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv preprint arXiv:2302.06706 , 2023. Previous versions accepted "Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022" with title "Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change)".

[WDKM21] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021.

[WJK08] Chenggang Wang, Saket Joshi, and Roni Khardon. First order decision diagrams for relational mdps. Journal of Artificial Intelligence Research , 31:431-472, 2008.

[WK10] Chenggang Wang and Roni Khardon. Relational partially observable mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1153-1158, 2010.

[WPY05] Jason D Williams, Pascal Poupart, and Steve Young. Factored partially observable markov decision processes for dialogue management. In Proc. IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems , pages 76-82. Citeseer, 2005.

[YFG07] Sungwook Yoon, Alan Fern, and Robert Givan. FF-Replan: A baseline for probabilistic planning. In Proc. ICAPS 2007 , pages 352-360, 2007.

[YRT + 22] Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang, Alexander Clegg, Manolis Savva, Eric Undersander, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2022. https://aihabitat.org/challenge/2022/ , 2022.

[YZY + 23] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.

[ZC21] Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. In ACL/IJCNLP (Findings) , volume ACL/IJCNLP 2021 of Findings of ACL , pages 4202-4213. Association for Computational Linguistics, 2021.

[ZYP + 22] Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, and Joyce Chai. DANLI: deliberative agent for following natural language instructions. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 1280-1298. Association for Computational Linguistics, 2022.

A Supplementary Material

This Supplementary Material section provides additional details and discussions to complement our main paper. In section B, we expand on the discussion of our approach. Section C presents specific results for the seven tasks of the ALFRED challenge as well as new tasks that can be solved in the environment. Section D provides a formal definition of Object-Oriented Partially Observable Markov Decision Processes (OO-POMDPs), while section E provides a detailed background for our planning approach. Finally, section F provides an in-depth explanation of our method for the ALFRED challenge.

B Further Discussion

In section 9, we touched on various aspects of our methodology. This section provides more detail on these areas. Our approach to the ALFRED challenge hinges on egocentric planning and a user-defined symbolic planning model. We refer to these models as intentional, as they are purposefully crafted by the user, integrating specific environmental objects and properties. Symbolic models function as a form of world model that facilitates planning. Some methods seek to learn a latent representation, but this often comes with a high sample complexity (HPBL23). Modular meta-learning can reduce the sample complexity (ALPK18, ORCC21, MvSE22). If cost is not a constraint, world models can become components of efficient strategies trained to solve problems in specific domains (SAH + 20).

However, with limited resources or data, generalizing to new situations can be challenging. Take an embodied agent in an environment (def D.8) with N skills and M types of objects, tasked with carrying out a sequence of tasks that require plans involving a new combination of k skills and m types of objects, where n ≪ N and m ≪ M 3 . Such setting is within the scope of methods like metalearning or continual learning for adapting the latent representation for such unfamiliar environments (FAL17). The priors in some modular meta-learning approaches are related with our notions of objects and actions, but their inference algorithms might not scale well for long plans or ignore that some users might also want intentional modelling (ALPK18, ORCC21, MvSE22). Egocentric planning, on the other hand, may provide predictable scalability in these scenarios. While the definition of open-world generalization might be more elusive, our method leans towards systematic generalization across actions and object types.

Planning using certain latent representations or world models can also prove costly. As explained by (AKFM22) (LATPLAN), not all factored representations afford the same level of scalability. The authors of LATPLAN apply variational autoencoders to learn a discrete latent representation, incorporating additional priors so the representation can be interpreted as a STRIPS symbolic model, the most basic type of symbolic model we use for egocentric planning. Symbolic models like STRIPS enable solvers with greater scalability and generalization.

An alternative approach involves methods that learn a symbolic planning model (AFP + 18, CDVA + 22). Object-oriented POMDPs are especially suitable as the grounding on objects and action labels allows for more efficient algorithms (CMW13, MCRW09), an advantage not enjoyed by the authors of LATPLAN as they only use pairs of images as training data. Learning a symbolic model helps manage the risk of compounding errors since the model can be debugged and improved. Real-world applications may require monitoring the agent's behaviour for specific objects or actions. Furthermore, specific actions may carry inherent risks that need to be avoided. In general, we anticipate that developers of embodied agents will use as much learning as possible, as much intentional modelling as necessary, and symbols to blend the two of them (KSV + 22).

Further integration between symbolic planning models and other ML models might help to reduce some limitations of our approach. For instance, in ALFRED, searching for a pan should make the agent head toward the kitchen. If another model like an LLM finds it likely that the pan is in the kitchen, we can use the scores to produce action cost in a similar way to the integration done in SayCan, (iBC + 23).

Egocentric planning connects to a body of work aimed at tackling expressive planning problems by harnessing the scalability of classical planners. While egocentric planning focuses on the partial observability of objects, (APG09, PG09) consider partial observability given a set of possible initial

states and deterministic actions, and (MMB14) considers full observability but with nondeterministic actions. Though these strategies can provide complete solutions proven offline, they can also serve as approximate methods for solving online problems. When planning with probabilistic effects, replanning assuming randomized determinization of actions can be a strong approach if there are no dead ends (YFG07).

Our symbolic planning models are extensions of STRIPS, but other options exist. Hierarchical task networks (HTN) (NAI + 03, GA14), for example, offers an alternative to STRIPS as they allow the specification of plans at various abstraction levels. Sometimes, it is more practical to outline potential plans using formalisms like linear temporal logic (LTL) formulae (DV13). Nonetheless, the relationship between these models has been examined, and it is feasible to integrate them with STRIPS-based planning (CC04, HBB + 19, MH13).

C Results in ALFRED's Task and New Tasks

C.1 ALFRED Task Performance

To analyze the strengths and weaknesses of our approach in different types of tasks, we conducted further evaluations by task type as in Table 5. Our findings reveal several noteworthy observations. Firstly, the success rates (SR) and goal completion rates (GC) for the "Pick_two_obj_and_place" task are lower than the rest, mostly due to these tasks involving more objects and with smaller masks. The task "look_at_obj_in_light" has the best performance due to the variety of lamp is low compare to other common objects. Secondly, there is no strong correlation between the number of interactions involved in a task type and its performance (STG + 20). For example, the task "pick_cool_then_place_in_receptacle" has an extra cooling step compare to "pick_and_place_simple" but still have better performance. A detailed description of each task can be found in the original ALFRED paper on page 16 to 19 (STG + 20).

C.2 New Task Performance

In section 8.4, we discuss the generalization of our approach to new tasks. The results are summarized in table 6.

Our approach enable us to reuse the definition of existing models to achieve zero-shot generalization to new tasks as long as the set of actions, objects and their relationships remain the same. In many embodied agent applications, agent need to have ability to adapt to new tasks. For instance, organizing a living room could involve storing loose items on a shelf or arranging them on a table. An

agent familiar with common household objects and equipped with the skills to handle them should seamlessly manage both types of cleaning tasks. However, transfer learning for neural networks remains challenging in the context of embodied agent tasks generalizing across objects and skills. Similarly, the template system used in FILM demands hand-crafted policies for new task types. The ALFRED dataset include instructions for each tasks that (STG + 20), although our approach does not use them. In a our egocentric planning formulation, a task can be broken down into a set of goal conditions that comprise objects of interest. The planner can then autonomously generate an action sequence to reach these goals. Consequently, our method can independently execute these tasks without the need for transfer learning or the addition of new task templates. To demonstrate this, we devised 5 new task types that were not present in the training data. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made to minimize perception errors. These tasks share the same world model as ALFRED and thus do not necessitate any additional modification beyond specifying a new goal. We selected environments and layouts from the training set where subgoals required for each new task type are feasible, ensuring the feasibility of the new task for our agent. Table 6 lists these new tasks and their success rates. We selected 20 scenarios for each new task and maintained the maximum 10 interaction error as in the ALFRED problem. Our method was able to achieve an impressive overall success rate of 82% without any additional adjustments beyond the new goal specification. However, due to the constraints of the ALFRED simulator, we're unable to experiment with introducing new objects and action types beyond what's available. For instance, "heating" in ALFRED is solely accomplished via a microwave. In real-world scenarios, we could easily incorporate ovens and actions to operate them with minor alterations to our PDDL domains. Similarly, we could introduce time and resource restrictions for each action, which are common in many PDDL benchmarks. The ability to accommodate new goals and constraints is a key advantage of our planning-based approach.

C.3 Computation

Perception and Language module was fine-tuned on a Nvidia 3080. The planning algorithm uses a 8 core Intel i7 CPU with 32 GB of ram. More detailed on traning can be found in (MCR + 22)

D Embodied Agents Generalizing within given Objects Types

In this section, we provide further details about embodied agents presented in section 4. Embodied agents perceive and act in environments to accomplish tasks. Navigating challenges and constraints imposed by their environment and tasks, these agents ensure safety and adhere to physical restrictions. Taskable agents rely on their surroundings and inherent capabilities, creating a strong dependency and often specializing in particular environments. This specialization shapes their cognition and problem-solving strategies, presenting an opportunity for robust agent design.

We exploit this limitation by focusing on generalization across tasks involving the same object types and their relationships, applicable to both physical and software agents. Taskable agents possess specialized capabilities for these object types, as seen in robotics where battery level, body state, and actuators are distinct variables (CST + 22). Object types also serve as inputs/outputs in end-toend approaches with learned representations and appear in embodied agents operating in software environments, using APIs and maintaining internal states. Our symbolic planning approach aims to provide a basis for designing adaptable and flexible agents that sense and act upon given object types. We start by defining object types, and entities grounded in these types.

Definition D.1 (Objects and Object Types) . An Object Type , denoted as ⟨T , V⟩ , consist of a finite set of object types T , along with a mapping V : t i → V i that assigns each type t i ∈ T a set of possible values V i . An Object o = ( t i , v ) is an instance of a type with a specific value, i.e., t i ∈ T and v ∈ V ( t i ) .

Example D.2 (Object types and Values) . An example of an object type is t objectSeen with values V ( x,y ) ×V direction ×V objectSubtype ×V id where V ( x,y ) is position in a grid, V direction is the direction the object is facing, V objectSubtype is the category of the object observed, and V id is a unique identifier for the object. Other related examples of object types are t movable with values V movableSubtype ×V id × V objectproperties , and t obstacle with values V obstacleSubtype ×V id ×V objectproperties . Furthermore, an action pickup with parameters Φ E ( a ) = ( t objectSeen ) is an action that picks up a specific object.

Object types are used as parameters of entities of actions of Object-oriented POMDP defined, and as parameters of actions and predicates in planning domains in section E.

Definition D.3 (Typed Entity) . Given a set of entities Ξ , object types ⟨T , V⟩ , and a mapping Φ : ξ → ⋃ k ∈ N T k . A Type Entity ξ ∈ Ξ is mapped to Φ( ξ ) = ( t 1 , t 2 , ..., t k ξ ) , with each t i ∈ T . This mapping associates each entity with a k ξ -tuple of object types, corresponding to k ξ parameters of the entity, defining the arity of the entity ξ . We refer to the variables var ∈ { var 1 , var 2 , ..., var k ξ } as the positional parameters of the entity k ξ , denoted ξ ( var 1 , var 2 , ..., var k ) .

Definition D.4 (Grounded Entity) . Given a set of typed entities Ξ , with object types ⟨T , V⟩ , and a mapping Φ , a Grounded Entity , denoted as ξ g for ξ ∈ Ξ , is a parametric object where each type argument of ξ specified by Φ( ξ ) is replaced by an object of the corresponding type. Specifically, ξ g = ξ ( o 1 , o 2 , ..., o k ξ ) , where each o i = ( t i , v i ) with v i ∈ V ( t i ) . The set of all grounded entities is denoted by Ξ g .

Now we define the environment where the embodied agent operates as a Partially Observable Markov Decision Process (POMDP) with costs. Cost POMDPs are related to classical MDPs but feature partial observability and use costs instead of rewards (MK12, GB13).

The literature sometimes confound the environment with the underlying the POMDP, but we want to make explicit what is available to the agent. Thus, while our underlying model is a POMDP, we consider agents that can aim to generalize without being fully aware of the distribution of future tasks, except for the actions and object types that are explicit in the environment. Therefore, we abuse slightly the notation to annotate with the objects defined below with the environment, denoted E .

Definition D.5 (Cost POMDP) . A Cost POMDP is a tuple ⟨ S E , A E , T E , C E , P E , Z E ⟩ , where S E is a set of states, A E is a finite set of actions, T E : S E × A E × S E → [0 , 1] is the transition function, C E : S E × A E → R > 0 is the cost function, P E is a set of perceptions, and Z E : S E × A E × P E × S E → [0 , 1] is the sensor function.

Object-oriented POMDP is an extension of a Cost POMDP that includes objects and their properties to be used for expressing perceptions and actions.

Definition D.6 (Object-oriented POMDP) . An Object-oriented POMDP Γ , represented by ⟨ S E , A E , Φ E , T E , C E , P E , Z E , T E , V E ⟩ , extends a Cost POMDP by incorporating a predefined set of Object Types ⟨T E , V E ⟩ . It defines the set of perceptions P E where a perception p ∈ P E is a finite set of objects { o 1 , o 2 , . . . , o m } , and the set of typed actions A E defined for the object types ⟨T E , V E ⟩ and the map Φ E .

In this context, Grounded Actions refer to grounded entities, where the set of entities is the actions A E . Values in object-oriented POMDPs can be dense signals like images or sparse signals like object properties. Values can have any structure, so they can refer to values in the domain of other types. Indeed, Object-oriented POMDPs can express Relational POMDPs (WK10, WJK08, SB09, TGD04, WPY05, DNR + 20). However, our approach emphasizes the compositionality of the objects and their properties.

Agents interact with the POMDP through an environment. Given a task, we initialize the environment using the task details. The agent receives the tasks and acts until it achieves the goal or fails due to some constraint on the environment. The agent achieves a goal if, after a sequence of actions, the environment's hidden state satisfies the goal.

Definition D.7 (Task) . Given an Object-oriented POMDP Γ , a Task is a tuple T E = ⟨ I E , G E ⟩ , where I E is an object that the environment interpret for setting the initial state, and G E is a set of goal conditions expressed as objects { g 1 , g 2 , . . . , g n } , with each g i being an instance of an object.

Definition D.8 (Object-oriented Environment) . Given a hidden Object-oriented POMDP Γ , an Objectoriented Environment , or just Environment , is a tuple E = ⟨ A E , T E , V E , reset , step ⟩ , where A E , T E , and V E are as previously defined, reset : T E → P E is a function that takes a task T E = ⟨ I E , G E ⟩ , resets the environment's internal state using I E and returns an initial observation, and step : A g E → P E × R is the interaction function that updates the environment's internal state according to the execution of a grounded action a g , for a ∈ A E , and returns an observation and a cost.

In some environments, I E might be opaque for the agent. In others, it might containg information not used by the environment like user's preferences about how to complete a task. However, we abuse this notion and refer to I E as the initial state of the environment.

Separating Object-oriented POMDPs from environments allows us to consider the scope of the generalization of our agents within the same POMDP. While the agent cannot see the state space directly, it is aware of the actions and the object types.

Definition D.9 (Embodied Agent) . Given an environment E = ⟨ A E , T E , V E , reset , step ⟩ , and a task T E = ⟨ I E , G E ⟩ , an Embodied Agent is a tuple ⟨ M I , π M , M U , G M ⟩ , where M I : P E × I E × G E →M is a function that takes an initial observation, a task, and returns the initial internal state of the agent, π M : M→ A g E is the agent's policy function that maps mental states to a grounded action in A g E , M U : M× A g E × P E × R →M is the mental state update function receiving a grounded executed action, the resulting observation, and the cost, and G M : M→ [0 , 1] is the goal achievement belief function that returns the belief that the agent has achieved the goal based on its mental state M .

Algorithm 2 provides a general framework for an agent's execution process. By instantiating the policy, mental state update function, and goal achievement belief function using appropriate techniques, the agent can adapt to a wide range of environments and tasks. Although new tasks can set the agent in unseen regions of the state space S E , the agent remains grounded on the known types and the actions parameterized on object types, T E . They offer opportunities for generalization by narrowing down the space of possible policies and enabling compositionality. The agent's mental state M can include its belief state, internal knowledge, or other structures used to make decisions. The policy π M can be a deterministic or stochastic function, learned from data or user-specified, or a combination of both. As the agent has an internal state, its policy can rely on a compact state or on the history of interactions.

Algorithm 2 Agent Execution Input: Environment ⟨ A E , T E , V E , reset , step ⟩ Input: Agent ⟨ M I , π M , M U , G M ⟩ Input: Task ⟨ I E , G E ⟩ Input: Probability threshold θ Output: Successful trace τ or Failure 1: p ← reset ( I E , G E ) 2: τ = [] ▷ Empty trace 3: M ← M I ( p, I E , G E ) 4: while G M ( M ) < θ do 5: a g = a ( o 1 , o 2 , ..., o k a ) ∼ π M ( M,A g E ) ▷ where a in A E and o i are objects. 6: τ. append ( a g ) 7: p ′ , c ← step ( a g ) 8: if failed ∈ p ′ then 9: return Failure 10: M ← M U ( M,a g , p ′ , c ) 11: return τ

The agent starts in the initial state set from I E and aims to achieve the goal G E . While the agent does not know the true state of the environment, agents can aim to generalize across actions and the object types as they are always available. While we prefer policies with lower expected cost, our main concern is achieving the goal. It is possible that a task might be unsolvable, possibly because of actions taken by the agent. We assume that failures can be detected, for instance, by receiving a special observation type failed . We further assume that after failure, costs are infinite, and failed is always observed.

The Iterative Exploration Replanning (IER), Algorithm 1 in section 5, is a instance of algorithm 2. Instead of using a parameter θ , it considers the problem solved when it obtains a plan that achieves the goal.

E Detailed Background: Parameterized Full-Observable Symbolic Planning

In this section, we describe the background definitions of parameterized full-observable symbolic models mentioned in section 5.1. We define planning domains and problems assuming full observability, including classical planning that further assumes deterministic actions, also known as STRIPS (GNT04, GB13). Planning domains and problems rely on typed objects (Def D.1), as object-oriented POMDPs, but planning actions include a model of the preconditions and effects.

Definition E.1 (Parametric Full-Observable Planning Domain) . A parametric full-observable planning domain is a tuple PD = ⟨T , V , P , A , Φ ⟩ , where ⟨T , V⟩ is a set of object types, P is a set of typed predicates and A is a set of typed actions, both typed by the object types and the map Φ . Each action a has an associated PRE a , expressing the preconditions of the action, and EFF a , expressing its effects. For each grounded action a ( o 1 , o 2 , ..., o k ) ∈ A g , the applicability of the grounded action in a state s is determined by checking whether the precondition PRE a is satisfied in s . The resulting state after applying the grounded action is obtained by updating s according to the effect update EFF a . The specification of EFF a is described below.

While the values in both parametric full-observable planning domains and object-oriented POMDPs can be infinite (def D.6), the values used in concrete planning problems are assumed to be finite. As a consequence, the set of grounded actions of a planning problems is finite (def D.4).

Definition E.2 (Parametric Full-Observable Planning Problem) . A parametric full-observable planning problem is a tuple ⟨PD , O , I, G ⟩ , where PD = ⟨T , V , P , A , Φ ⟩ is a parametric full-observable planning domain, O is a finite set of objects, each associated with a type in T , I is a description of the initial state specifying the truth value of grounded predicates in P g , G is a description of the goal specifying the desired truth values of grounded predicates in P g , where P g are the predicates P grounded on object types ⟨T , V⟩ and the map Φ encoding their typed arguments.

This definition covers a wide range of symbolic planning problems featuring full observability including probabilistic effects (KHW95). In particular, the definition can be specialized for classical planning problems, where the actions are deterministic, corresponding to STRIPS, to the most studied symbolic model (GB13).

For classical planning, given an action a ∈ A parameterized by a list of object types a ( t 1 , t 2 , ..., t k ) , their precondition or effect are formulas γ expresse in terms of the variables var ∈ { var 1 , var 2 , ..., var k } that refer to positional parameters of the action. We denote the grounded formula γ ( o 1 , o 2 , . . . , o k ) as γ with each variable var i that occurs in γ is replaced by corresponding objects o i .

Definition E.3 (Parametric Classical Planning Domain) . A parametric classical planning domain is a parametric full-observable planning domain where each action a ∈ A has preconditions PRE a , and an effect update EFF a represented as a tuple ( ADD a , DEL a ) of add and delete effects, all sets of predicates in P . Given a grounded action a ( o arg ) with o arg = o 1 , o 2 , . . . , o k , the precondition PRE a is satisfied in a state s if PRE a ( o arg ) ⊆ s . The effect update EFF a describes the resulting state as ( s \ DEL a ( o arg )) ∪ ADD a ( o arg ) after applying the action.

Definition E.4 (Parametric Classical Planning Problem) . Given a parametric classical planning domain PD , a parametric classical planning problem is a parametric full-observable planning problem where I and G are conjunctions of predicates in P (equivalently viewed as sets).

As Egocentric Planning relies on replanning over a sequence of related planning problems, it is convenient to define some operations required for exploration, as the agent discovers more objects. They allow us to implement the function ExploreAct in Algorithm 1.

Definition E.5 (Precondition Extension) . Given a parametric full-observable planning problem and an action a ∈ A , extending the precondition PRE a with ∆ PRE a , denoted PRE a ⊕ ∆ PRE a , is a new precondition that is satisfied in a state s if PRE a and ∆ PRE a are both satisfied in s .

Definition E.6 (Effect Extension) . Given a parametric full-observable planning problem and an action a ∈ A , extending the effect EFF a with (∆ ADD a , ∆ DEL a ) , denoted EFF a ⊕ (∆ ADD a , ∆ DEL a ) , is a new effect that applied to a state s results the new state s ′ that is like s but with modifications EFF a , then ∆ DEL a is not true in s ′ , and ∆ ADD a is true in s ′ , in that order, assuming that neither ∆ DEL a nor ∆ ADD a appear in EFF a .

With these operations we can define the Egocentric Planning algorithm for parametric full-observable planning problems. For instance, in the case of probabilistic planning the preconditions can be defined as in classical planning, so the precondition update is the same. The effect update of probabilistic actions is a probability distribution over the possible effects. However, extending the effect with a new effect is straightforward as the new effect must enforce that Pr ( ADD a | s ) = 1 and Pr ( DEL a | s ) = 0 .

For classical planning, the precondition extension is the union of the original precondition and the new precondition as both are a set of predicates. The EFF a of classical planning actions ( ADD a , DEL a ) is a tuple of sets of predicates, so updating the effect amounts to updating both sets of predicates.

Definition E.7 (Precondition Extension Classical) . For parametric classical planning problems, extending the precondition PRE a ⊕ ∆ PRE a is PRE a ∪ ∆ PRE a .

Definition E.8 (Effect Update Classical) . For parametric classical planning problems, an update to the effect ( ADD a , DEL a ) with (∆ ADD a , ∆ DEL a ) is ( ADD a ∪ ∆ ADD a , DEL a ∪ ∆ DEL a ) .

For ALFRED, as the planning domain in classical, ExploreAct ( a , o ) creates a copy of a extended with a new precondition (unknown o ) so a can only be applied if o is still unknown. The effects are extended by deleting (unknown o ) and adding (explored) , so applying the exploration action satisfies the goal (explored) .

F Approach: Additional Details

In this section, we provide additional details on our approach discussed in section 6. Both the environment and the symbolic planning model consider objects and parametric actions, but they are represented differently.

For instance, our approach involves a vision model that perceives in a scene objects of known classes, such as apple or fridge , including their depths and locations (see sections 4 and D). Similarly, the planning domain includes corresponding object types, such as AppleType and FridgeType (see sections 5.1 and E). When the agent perceives an apple at location l1 , the algorithm creates a fresh object o1 for the apple, sets its type to (objectType o1 AppleType) , and its location to (objectAtLocation o1 l1) . Additionally, the domain contains information about what can be done with the apple, such as (canCool FridgeType AppleType) . Likewise, while the ALFRED environment contains actions like toggle , the planning domain contains actions like coolObject , which is parameterized on arguments such as the location, the concrete object to be cooled, and the concrete fridge object. Finally, if the task provided in natural language requests to cool down an apple, the planning problem goal contains, for instance, (exists (isCooled ?o)) , which is satisfied when some object has been cooled down.

The remainder of this section is structured as follows. First, we provide additional details on the Language and Vision modules, including the specific classes they detect and how they are utilized. Next, we elaborate on the semantic spatial graph and its role in generating the agent's mental model, which tracks object locations as the agent perceives and modifies the environment. We then provide specific details on the planning domain used for ALFRED, which completes the elements used in our architecture depicted in Figure 1. Finally, we conclude the section with additional details on the egocentric planning agent that won the ALFRED challenge.

F.1 Language and Vision Module

F.1.1 Language Module

We used a BERT-based language models to convert a structured sentence into goal conditions for PDDL (DCLT19). Since the task types and object types of defined in the ALFRED metadata, we use a model to classify the type of task given a high-level language task description listed in Table 9

and a separate model for deciding objects and their state based on the output of the second model in Table 7 and Table 8. For example, given the task description of "put an apple on the kitchen table," the first model will tell the agent it needs to produce a "pick-up-and-place" task. The second model will then predict the objects and receptacle we need to manipulate. Then the goal condition will be extracted by Tarski and turn in to a PDDL goal like (on apple table) using the original task description and output of the both models. We use the pre-trained model provided by FILM and convert the template-based result into PDDL goal conditions suitable for our planner. Although ALFRED provides step-by-step instructions, we only use the top-level task description since they are sufficient to predicate our goal condition. Also, our agent needs to act based on the policy planner, which makes step-by-step instructions unnecessary. This adjustment could also make the agent more autonomous since step-by-step instructions are often not available in many embodied agent settings.

F.1.2 Vision Module

The vision module is used to ground relevant object information into symbols that can be processed by our planning module. At each time step, the module receives an egocentric image of the environment. The image is then processed into a depth map using a U-Net, and object maks using Mask-RCNN. (HGDG17, LCQ + 18). Both masks are WxH matrices where W, and N are the width and height of the input image. Each point in the depth mask represents the predicted pixel distance between the agent and the scene. The average depth of each object is calculated using the average of the sum of the point-wise product of the object and the depth mask. Only objects with an average reachable distance smaller than 1.5 meter is stored. The confidence score of which our agent can act on an object is calculated using the sum of their mask. This score gives our egocentric planner a way to prioritize which object to interact with.

F.2 Semantic Spatial Graph

The spatial graph plays the role of an agent's memory during exploration and bridges the gap between grounded objects and the planner's state representation requirements. The exploration employs an egocentric agent who only has direct access to its local space where action can be executed within a distance threshold of 1.5 meters. The key for each node is encoded as location, with visual observations being the values. The edges represent the agent's actions, comprising four specific actions: MOVEAHEAD, MOVEBACK, TURNLEFT, and TURNRIGHT, with movement actions having a step size of 0.25 meter and turning actions featuring a turn angle of 90 ◦ . This spatial graph is updated and expanded over time through policies generated by our egocentric planner, with each node storing object classes, segmentation masks, and depth information produced via the vision module. Fig. 2 shows an example of a semantic spatial graph.

Non-Object Nodes

Object Nodes

The spatial graph undergoes continuous updates and expansions as a result of the policies generated by our egocentric planner. It involves the extraction and storage of objects within the graph, subject to a confidence score threshold of 0.8. Objects that meet or exceed this confidence score are deemed reliable and considered for storage. Each node within the graph maintains a record of various attributes associated with the objects, including object classes, segmentation masks, and depth information. The object classes provide information about the category or type of each object, allowing for efficient categorization and analysis. The segmentation masks of the extracted objects are stored as a list of pixel coordinates. These masks outline the boundaries of each object within the captured visual data. By storing the masks in this format, we can later retrieve and utilize the spatial information of each object during subsequent analyses or interactions. Moreover, the distance to each object is calculated by leveraging the predicted distance of the top 100 pixels with reference to the depth mask generated by the U-net model. The U-net model is responsible for producing a depth mask that estimates the distances of various points within the scene. By employing the predicted distance values of the top 100 pixels associated with each object, we obtain an approximate measure of its distance from the egocentric viewpoint. Actionable objects are treated as affordances at each node, allowing the agent to perform desired actions. Nodes with objects of interest are ranked based on the confidence scores outlined earlier output by the MaskRCNN.

F.3 Egocentric Algorithm for ALFRED

The egocentric planning algorithm for ALFRED uses location as the only anchor types T a , and the move action as the only exploration action X . Consequently, exploration plans employ move actions to visit unexplored locations, revealing new objects visible from such locations. As the agent explores, the mental model creates new objects and adds them to the initial state, facilitating the incremental construction of a semantic spatial graph, which maps the environment and object locations. An ALFRED task, T E = ⟨ I E , G E ⟩ , is characterized by a goal that corresponds to one of the seven tasks detailed in section 2, and involves specific predicates for the task-related objects. The goal is expressed as an existential formula over potential objects, as it refers to objects not visible from the initial state (Listings 2).

We also implemented some ALFRED-specific optimizations. For example, we mitigate the risk of executing irreversible actions by only attempting them after devising a plan projected to achieve the goal. This strategy implies that some actions are unavailable during the exploration phase. For each time step, our egocentric planner will produce a goal or exploration action depending on whether the agent has sufficient information to achieve its goal. The replanning loop repeats when an action fails. Priority are given to unknown location based on objects of interest for a particular task and number of total objects found in an area. They are ranked base on number of such objects. Information are gathered for each action taken which is stored in the semantic map. In order to reduce the amount of overall exploration required at the online planning stage, we allow our agent to first conduct 500 random exploration movements. The semantic graph after exploration is used as the initial state for our egocentric planning setup. To enhance efficiency, we tailor the planning domain to the specific task, disregarding actions unlikely to contribute to reasonable plans to achieve the goal. This optimization can be automatically implemented via reachability analysis, a technique commonly used in classical planning to prevent grounding actions irrelevant to achieving the goal from the initial state (Hof01).

F.4 PDDL Example

Below we provide examples assuming that the natural language description of the task is "heat potato in the microwave and put the potato on the table." We start by generating the planning setup for the egocentric planner, given the task identified as "pick-heat-then-place-in-recep." The objects involved in this manipulation are classified as "MicrowaveType", "PotatoType" and "TableType". The egocentric planner will eventually produce a sequence of actions to accomplish the desired task. In this case, the steps might include picking up the potato, heating it, and then placing it on the table.

Listings 1 and 2 show the PDDL domain and problem files for the example task. In a PDDL domain for classical planning, both predicates and actions arguments are expressed as a list of arguments, each one a variable with an optional type. For instance, (conn ?x1 - location ?x2 -location ?x3 movement) express a predicate conn with three arguments: two locations and a kind movement: ahead, rotate-left or rotate-right. In a PDDL problem for classical planning, object constants are associated with an optional type, and the initial state is a list of predicates grounded on existing constants. For instance, l1 l2 l3 location means three constants of type location . The initial state could be (and (at l1) (conn l1 l2 ahead)) . An action (MoveAgent ?from ?to -location ?dir -movement) has as a precondition that the agent is at the location ?from and both locations are contiguous in the direction ?dir . Executing the action sets to false that the agent is at the location ?from , and sets to true that the agent is at the location ?to . Classical plans in PDDL are sequences of actions grounded on constants. For instance, a first possible action might be (MoveAgent l1 l2 ahead) , leading to the state (and (at l2) (conn l1 l2 ahead)) . For an introduction to PDDL see (HLMM19).

Next, Listings 2 shows the generated the planning problem based on the initial observation. This involves specifying the initial state of the environment and the goal that we want to achieve through the planning process.

As not plan was found for the problem in Listing 2, the goal is changed to exploration in listing 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Egocentric Planning, alternating exploration and task solving symbolic planning for long-horizon object-oriented POMDP in environments with deterministic action effects. The presented method is used as the planner in a hybrid agent, SOTA in the 2022 ALFRED benchmark, with neural perception. The agent is based on a previous successful design (FILM), with semantic SLAM in the latter replaced by a graph representing the current knowledge about the scene, besides the novel open loop planning algorithm exploiting this graph. The experimental results show a considerable improvement over FILM at the cost of longer trajectories required to succeed due to the time required to gather information about the scene, besides error analysis  and ablations. An exciting result is the generalization to new task types not present in training.\n\nSTRENGTHS:\n- The proposed method enables out-of-the-box generalization to new task types (within the same set of objects and relationships).\n- The graph representation and planner algorithm work as a tailored alternative to semantic SLAM for object-oriented POMDPs with remarkable performance.\n\nWEAKNESSES:\n- The presented approach is effective under the assumption of deterministic action effects, but in real-world usage this should not generally apply. The paper consequentially discusses some possible ways to overcome this limitation, but also claims some better performing alternatives should be considered as baselines rather than comprehensive solutions, since they are limited in scope. I'd encourage rephrasing that discussion, which as is could be interpreted as unfair to the alternatives.\n- The need for an exploration phase, when the egocentric planner is in principle designed to determine when to explore, seems a little ad hoc. I wish there was a clearer explanation why this cannot be provided by the main algorithm.\n\nQUESTIONS:\n1. How could the proposed framework be extended for multiple agents interacting with the environment?\n2. Related to one of the weaknesses, what is missing in the egocentric planner to be capable of successfully exploring the environment until enough information is gathered to solve the task?\n\nLIMITATIONS:\nI think the paper makes a good job describing limitations of the proposed method, but, as mentioned in Weaknesses, I think the judgment of better performing alternatives as baselines, given the acknowledged limitations of the proposed method, could be seen as unfair, so I think an improved contextualization is needed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach combining symbolic planning and object-oriented POMDPs for symbolic planning, which gets extremely strong performance on the ALFRED benchmark and won the CVPR ALFRED challenge. Their approach uses PDDL, but extends it with a set of exploration-focused actions. They use a combination of explicit knowledge and heuristics (exploring close to a seen object for example) that are learned from data. They explore for 500 steps, using this to build a spatial-semantic graph (instead of an explicit map like many previous methods). They then use this with an off-the-shelf task planner to choose which actions to execute.\n\nSTRENGTHS:\n- Building a spatial semantic graph seems like a better (more scalable) way of solving ALFRED tasks than explicit maps\n- Strong performance on a well-respected benchmark\n- A lot of great ideas, and some of the explanation is very good\n- Great to see a *new* approach to solving ALFRED, not just building off of FILM/HLSM\n\nWEAKNESSES:\n- The name \"egocentric planning\" doesn't make a lot of sense to me; everything in ALFRED is going to be some manner of egocentric planning\n- Not clear how general this is - lots of engineering in the planning domain. More analysis would help here; ideally the authors could run on a different domain (OVMM might be one - ovmm.github.io), but there really aren't good options with strong existing baselines like ALFRED. Instead maybe they could describe in more detail how it would be applied to other domains. \n- Writing could be improved. Several typos (Apporach --> Approach), IER (exploration algorithm) not being referenced by name in text.\n- It's not really clear this is a *learning* contribution - learning components seem minor. I think this is ok, because it's still a useful result on a learning problem, but I could see the argument against it.\n- Very engineering heavy.\n\nQUESTIONS:\n- How is the spatial graph constructed? It seems like a lot of the detail here is lacking.\n- How important are the 500 exploration steps? Seems like it should be able to work well without this, if it's really a good POMDP planner.\n- Construction of the initial state was also a bit unclear to me.\n\nLIMITATIONS:\n- Only applied to one benchmark\n- Need for 500 exploration steps at the beginning seems weird\n- Very dependent on perception models - perfect detection, depth, etc., which limits application to other domains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a modular approach that combines symbolic planning and object-centric Cost POMDP for solving ALFRED tasks. The proposed method demonstrates improvements compared to previous end-to-end and modular approaches, such as FiLM. Unlike methods like FILM, HLSM, and Prompter, EPA utilizes a semantic graph representation instead of a top-down 2D map.\n\nThe method incorporates an initial phase of 500 exploration steps to gather sufficient knowledge, which is crucial for determining an appropriate expandable initial state. Object information is selectively saved only when objects are within immediate reach (0.25m ahead). This selective saving facilitates the conversion of observations into a symbolic state and reduces the length of generated plans.\n\nThe authors' findings suggest that the performance enhancements achieved by EPA over FILM and similar approaches primarily stem from its iterative planning approach. This approach enables the agent to recover from failure scenarios through flexible subgoal ordering, leading to improved performance.\n\nAccording to the authors, EPA achieved the second-highest ranking on the ALFRED leaderboard, closely following the Prompter method. The superiority of Prompter, however, is attributed to modifications in obstacle size and reachable distance rather than the use of prompts.\n\nSTRENGTHS:\n1. The paper demonstrates the use of preconditions and effects through PDDL to improve the overall success of long-horizon tasks, especially unseen success rate. \n2. The paper combines the use of symbolic planning using learned vision and language models and highlights how certain aspects of generalization can be achieved by abstraction.\n\nWEAKNESSES:\n1. The current assumptions on semantic spatial graphs require random exploration for 500 steps to visit each location and form a node in the graph. The paper reports a drop in performance if this initial observation phase is ignored. This approach has two major limitations: (1) it dramatically increases the timestep overhead for the proposed agent as compared to the existing works. (2) it assumes a static unchanging environment after mapping and will likely fail in realistic environments with dynamic obstacles. Given the existing visual-inertial SLAM approaches, as noted by the authors, it seems that this issue can be mitigated. Some existing approaches for topological mapping [1] might also be relevant, and in turn, improve the path length weighted success rate (PLWSR).\n\n1. Writing PDDL domain definitions and problems for each task is known to be a tedious coding task. Any errors in representing the available objects and actions would yield no plan. The current approach seems too close to reverse engineering the process of creating trajectories for the ALFRED task.  While the authors report 150 hours for PDDL domain and problem definitions, a large chunk of the work involving object types and action predicates is already described in the ALFRED metadata. This does not give a reasonable perspective on how many hours would it take to scale and maintain this approach further, especially in the physical world.\n\n *[1] Chaplot, D.S., Salakhutdinov, R., Gupta, A. and Gupta, S. 2020. Neural Topological SLAM for Visual Navigation. In CVPR.*\n\nQUESTIONS:\n1. Could you kindly provide some clarification regarding the semantic spatial graph? Specifically, what information does it contain and what is the average size of such a graph? The description in Appendix F2 suggests that the graph incorporates visual observations, segmentation, and depth. I would like to understand if this graph represents the visual scene \"in front of\" the agent. If so, I have a couple of related questions:\n    1. In case the agent is at the same location but facing a different direction, would the information on the graph be overridden?\n    2. If not, does each node in the graph contain a representation of the \"360-degree visual observation\" at that particular location? How are the \"actions\" represented as edges in this context?\n2. I'm curious to know how the approach encodes common sense knowledge or visual knowledge (mentioned in Lines 57-58) as part of the domain definition in PDDL.\n3. Could you please explain how the \"exploration actions\" are defined? In Appendix F3, Listing 3, it is mentioned that the agent cannot hold something and explore the environment. For example, can the agent actively search for the coffee after picking up the cup? I would appreciate some clarification on this.\n4. I would like to understand how the possible predicates are listed in the PDDL. Additionally, how are the preconditions and effects identified for each predicate? Are they learned or inferred from visual observations, language goals, or the agent's interaction to gather information? Are these predicates hard-coded by a human? If so, I'm interested in understanding how this differs from the classical task planning setup.\n5. It is not entirely clear how the generalization to new tasks is achieved, particularly when the language module is trained to output a task type out of seven tasks (as mentioned in lines 73-74). Could you please elaborate on the meaning of the statement in lines 308-309, \"Our egocentric planning breaks a task into a set of goal states, autonomously generating an action sequence\"?\n\nLIMITATIONS:\nOverall, it appears that the current approach is heavily focused on reverse engineering the trajectory creation process for the ALFRED task, utilizing learned vision and language models. While the method demonstrates a high success rate on the ALFRED benchmark for unseen scenarios, there is room for improvement in terms of clarity and the overall significance of the proposed approach. It remains uncertain how applicable the use of semantic spatial graphs or PDDL domain definitions would be beyond the ALFRED simulated benchmark, particularly when considering real-world physical environments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of embodied tasks in which the agent needs to plan over long task horizons given natural language instruction. Current methods that use end-to-end training leads to entangled representation, which makes it hard to solve the task. On the other hand, planning methods such as PDDL can produce high-quality actions given a well-defined problem specification. To solve the task, the paper proposes a method that consists of two parts: (1) goal-oriented exploration that aims to gather missing information, and (2) a classical planning method that aims to produce a feasible action. The proposed method is evaluated on the embodied benchmark ALFRED. The method improves the prior SOTA performance by 8.3%, winning the ALFRED challenge at CVPR 2022 Embodied AI workshop.\n\nTo be more precise, the proposed method consists of several parts: (1) a visual module for semantic segmentation and depth estimation of the scene, (2) an egocentric planner for planning given the information gather, (3) a semantic spatial graph for scene memorization. At the beginning of the task, the agent is provided with 500 steps to explore its surroundings. After that, the information is converted to a semantic spatial graph for the input of PDDL. Figure 1 shows the method. The algorithm inerates using the following steps: (1) find a path for reaching the goal, (2) if the goal is not reached, do exploration again\n\nTable 1 shows the main result of the paper. And the rest of the experiments section provides a detailed ablation study of the method.\n\nSTRENGTHS:\n1. The writing of the paper is clear and easy to follow. For instance, in the introduction, I can easily understand the motivation of the method.\n\n2. The proposed method is a winning approach at the CVPR Embodied AI workshop in 2022. This shows that the method has been thoroughly tested, and the result is convincing.\n\n3. In the AI era, it is nice to see a classical planning method is robust in such tasks over a neural network-based method.\n\n4. The related work is reasonable. It covers important papers such as Saycan\n\n5. Overall, I think this is a good paper, and the method is elegant and promising.\n\nWEAKNESSES:\nThe proposed method could be very specific to the ALFRED task, in the sense that the method is solely optimized and engineered for the  ALFRED task. For example, in some tasks, there is no abstraction of the action space, like OpenDrawer.\n\nQUESTIONS:\n1. If I want to deploy such a method in the real world, what would be the challenge and the additional step to do this?\n\n2. Following the previous questions, what is the Sim-to-real gap here?\n\n3. Will the method be able to handle tasks such as you need to open the drawer to find the coffee mug in it, instead of target objects being visible to the agent? I will say this is more challenging in the POMDP setting than the setting in the paper.\n\n4. What are some failure cases? Could you provide a couple of examples?\n\n5. Is it possible to integrate such a method with semantic exploration (https://devendrachaplot.github.io/projects/semantic-exploration)?\n\nLIMITATIONS:\nThe paper has clearly stated the limitations of the paper, including the method cannot recover from irreversible failures, being sensitive to perception errors, and having no memory for belief tracking.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a hybrid approach leveraging neural perception models and symbolic planners for egocentric planning and task completion in embodied environments. They demonstrate their approach in ALFRED benchmark winning the 2022 CVPR challenge. Their central idea is the use of symbolic planners, which are typically used in fully observable settings. To deal with partial observability, they perform iterative exploration with symbolic planning. Structurally, their implementation is similar to the previous SOTA on ALFRED — FILM. However, instead of using a semantic map as FILM does, they use a graph structure for object and location information storage. This graph is updated through exploration and then used by the downstream symbolic planner.\n\nSTRENGTHS:\n- The authors’ hybrid approach leveraging symbolic planners is novel amongst the modular approaches for long-horizon EAI tasks, and can be promising for future research.\n- The authors won the CVPR 2022 EAI ALFRED challenge.\n\nWEAKNESSES:\n- Poor clarity of exposition: I found the major aspects of authors’ technical approach poorly explained in the paper. This really limits reproducibility as well as use by the community IMO and consequently limits the value of the paper.\n    - Spatial graph: Unclear how the graph is built and what does it look like. The authors say that the graph encodes location as the node key and visual observations as values. However, it is unclear what is the co-ordinate system for location (consequently would we get the same graph for the same environment and task but different agent start position?), what happens when the agent visits the location more than once, and lastly, how is it used for the low-level policies that have to navigate given there is no map. \n    - Object-centric POMDP: The authors briefly mention the Cost POMDP but do not appropriately define or explain it at all in the main paper. Most definitions are pushed to supplementary. While it is okay to push nitty-gritty details to supplementary, without properly framing the problem and then grounding their approach in the framework, I am not sure if I can buy the authors’ claim that their approach is theoretically sound (L121).\n    - Exploration: Unclear how exploration is handled. The only description is in the intro (L52-56).\n    - The algorithm is not commented and contains symbols which are never defined or explained, making it difficult to understand what did the authors actually implement and why:\n        - Symbol Ge is never defined\n        - Unclear how cost c of action is computed and used in function $M^{PD}$.\n    - The authors say that the PDDL domain and environment definitions of actions are misaligned, which make sense (L149). However they never explain how this alignment is obtained in their approach. \n    - Furthermore, given the PDDL and planning terminologies might not be accessible/known to everyone, I encourage the authors to be more careful and clear in their descriptions. For instance, what exactly are anchor object types and why do we need them? Also unclear what other anchor object types can exist in EAI settings. \n    - I didn’t get much out of Sec.4 and 5 despite the fact that I work with symbolic planning and EAI and even when there is so much to explain about the approach as mentioned above. I recommend rewriting entire Sec.4 and 5.\n- Generalization: Authors claim and attempt to show generalization in Sec.8.4. However, this section had no details whatsoever about the new tasks they use nor a reference to supplementary. I managed to find some details in supplementary. However, I am sure a general reader will be lost here. Furthermore, the authors claim that they do better on the new tasks and achieve zero-shot success 82% times, better than other methods but do not show any such comparison in their work (tab.6 supplementary). Given this is one of the main claims of advantage of their approach, I’d like to see comparison with neural and template-based approachers, and also with the prompter method. \n\nWith the above (also see limitations section below), IMO the paper is not yet at par for publication at Neurips. In its current state, it can be a great workshop paper but requires major rewriting and additional comparisons e.g., for generalization claims otherwise.\n\nQUESTIONS:\n- Table1: main results should have bolded numbers and indications on which metrics need to be higher/lower for better task performance (e.g., with $\\uparrow$ and $\\downarrow$) to improve interpretability of results. Similarly sorting rows in Tab2 based on performance in unseen might be useful.\n- L166: “This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks” — not sure how?\n- Results: Given that the LGS-RPA also comes close in terms of GC and SR (unseen) to the authors approach, I encourage the authors to discuss LGS-RPA in the result section as well.\n- Tab:3 Unclear why the method only achieves ~60-70% GC with all ground truth available. Is this the upper bound? Why is the upper bound not 100% given that the env. is deterministic and we are using a symbolic planner?\n- Is the planning performed from scratch every time? I imagine that the state of the world and thus the graph have incremental changes so wondering if the authors do anything smart to reduce the planning time at each iteration.\n\nLIMITATIONS:\n- The authors do not touch upon the issue of obtaining PDDL domain in the first place, which is core to their approach. Unclear how this would scale for real world applications/agents and how errors in domain file can impact their approach. Lastly, they mention that they can handle additional constraints e.g., energy during planning. However, in practice, I’ve found planning time to blow up when using durative and cost constraints. Unclear again how authors approach would scale for more complex problems. Also, on that note, I’d like to see planner time numbers for the tasks perhaps in the supplementary. \n- Similarly, the authors say that they are more robust to perception errors however they say they do so using hand-crafted policies for replanning (Sec.8.3). Unclear if the robustness is because of these handcrafted policies or because of their iterative planning approach combining exploration and symbolic planning.\n- The  section on broader impact and societal impact is missing. I'd encourage the authors to think about real world applications that their work might enable for this section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}