SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaotian Liu*
ServiceNow Research Montreal, QC, Canada xiaotian.liu @mail.utoronto.ca
Hector Palacios†
ServiceNow Research Montreal, QC, Canada hectorpal@gmail.com
Christian Muise
Queen's University Kingston, ON, Canada christian.muise @queensu.ca
Embodied agents face significant challenges when tasked with performing actions in diverse environments, particularly in generalizing across object types and executing suitable actions to accomplish tasks. Furthermore, agents should exhibit robustness, minimizing the execution of illegal actions. In this work, we present Egocentric Planning, an innovative approach that combines symbolic planning and Object-oriented POMDPs to solve tasks in complex environments, harnessing existing models for visual perception and natural language processing. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability, achieving an impressive  $36.07\%$  unseen success rate in the ALFRED benchmark and winning the ALFRED challenge at CVPR Embodied AI workshop. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions, as well as what object types reveal information about others. It can naturally scale to solve new tasks beyond ALFRED, as long as they can be solved using the available skills. This work offers a solid baseline for studying end-to-end and hybrid methods that aim to generalize to new tasks, including recent approaches relying on LLMs, but often struggle to scale to long sequences of actions or produce robust plans for novel tasks.
Embodied task accomplishment requires an agent to process multi-modal information and plan over long task horizons. Recent advancements in deep learning (DL) models have made grounding visual and natural language information faster and more reliable  $(\mathrm{MBP}^{+}21)$ . As a result, embodied task-oriented agents have been the subject of growing interest  $(\mathrm{STG}^{+}20, \mathrm{YRT}^{+}22, \mathrm{WDKM}21)$ . Benchmarks such as the Action Learning From Realistic Environments and Directives (ALFRED) were proposed to test embodied agents' ability to act in an unknown environment and follow language instructions or task descriptions  $(\mathrm{STG}^{+}20)$ . The success of DL has led researchers to attempt end-to-end neural methods (ZC21,  $\mathrm{SGT}^{+}21$ ). In an environment like ALFRED, these methods are mostly framed as imitation learning, where neural networks are trained via expert trajectories. However, end-to-end optimization leads to entangled latent state representation where compositional and long-horizon tasks are challenging to solve. Other approaches use neural networks to ground visual information into persistent memory structures to store information  $(\mathrm{MCR}^{+}22, \mathrm{BPF}^{+}21)$ . These approaches rely on templates of existing tasks, making them difficult to generalize to new problems or unexpected action outcomes. ALFRED agents must navigate long horizons and skill assembly, operating within a deterministic environment that only changes due to the agent's actions.
*Work done when Xiaotian Liu was an intern at ServiceNow Research.
$^{\dagger}$ Work done when Hector Palacios was a research scientist at ServiceNow Research.
Long sequential decision problems with sparse rewards are notoriously difficult to train for gradient-based reinforcement learning (RL) agents. But a symbolic planner with a well-defined domain can produce action sequences composed of hundreds of actions in less than a fraction of a second for many tasks. Embodied agents might need to conduct high-level reasoning in domains with long action sequences. In such scenarios, human experts can opt to model the task within a lifted planning language. Planning languages such as Planning Domain Definition Language, or PDDL (HLMM19), is the most natural candidate, given the high scalability of planners that use such languages (GB13). For instance, we define the tasks of a home cooking robot with actions, such as opening, move-to, pick-up, put-on, and objects, such as desk, stove, egg found in common recipes.
However, most symbolic planning techniques are relatively slow in partially observable environments with a rich set of objects and actions. Thus we propose an online iterative approach that allows our agent to switch between exploration and plan execution. We explicitly model the process of actively gathering missing information by defining a set of unknown-objects and exploration-actions that reveals relevant information about an agent's goal. Instead of planning for possible contingencies, we use an off-the-shelf classical planner(Hof01) for fully-observable environments to determine whether the current state has enough knowledge to achieve the goal. If more information is needed, our agent will switch to a goal-oriented exploration to gather missing information. Our approach can seamlessly incorporate any exploration heuristics, whether they are learned or specified.
We implemented and evaluated our method on the popular embodied benchmark ALFRED using only natural language task descriptions. To demonstrate the effectiveness of our method, we use the same neural networks for both visual and language grounding used in FILM, which was the SOTA and winner of the 2021 ALFRED challenge  $(\mathrm{MCR}^{+}22)$ . See a comparison with other methods in Table 1. By replacing FILM's template-based policies with our egocentric iterative planner, our method improved FILM's success rate (SR) by  $8.3\%$  (or a  $30.0\%$  relative performance increase) in unseen environments winning the the ALFRED challenge at CVPR 2022 Embodied AI workshop. Our empirical results show that the performance increase was attributed to a more robust set of policies that account for goal-oriented exploration and action failures. We also show that our method can conduct zero-shot generalization for new tasks using objects and actions defined in the ALFRED setting.
2 ALFRED Challenge
ALFRED is a recognized benchmark for Embodied Instruction Following (EIF), focused on training agents to complete household tasks using natural language descriptions and first-person visual input. Using a predetermined set of actions, the agent operates in a virtual simulator, AI2Thor  $(\mathrm{KMG}^{+}17)$  to complete a user's task from one of seven specific classes instantiated in concrete objects. For example, a task description may be "Put a clean sponge on the table," and instructions that suggest steps to fulfill the task. The agent must conduct multiple steps, including navigation, interaction with objects, and manipulation of the environment. Agents must complete a task within 1000 steps with fewer than 10 action failures. We base our approach on task descriptions rather than the provided instructions. Task complexity varies, and an episode is deemed successful if all sub-tasks are accomplished within these constraints. Evaluation metrics are outlined in Section 8.
3 Overview of Egocentric Planning for ALFRED
Figure 1 illustrates our proposed method integrating several components to facilitate navigation and task completion: a visual module responsible for semantic segmentation and depth estimation, a language module for goal extraction, a semantic spatial graph for scene memorization, and an egocentric planner for planning and inference. Initially, high-level language task description is utilized to extract goal information, and the agent is provided with a random exploration budget of 500 steps to explore its surroundings. The random exploration step is used to generate diverse set of object cluster for further exploration using our planner. Subsequently, at  $t = 500$ , the gathered information from the semantic spatial graph is converted into a PDDL problem for the agent. We employ a novel open-loop replanning approach, powered by an off-the-shelf planner, to support further exploration and goal planning effectively.
Figure 1: Egocentric Planning for ALFRED
To facilitate egocentric planning, we first define the ontology of the planning problems, which encompasses action schemas, object types, and potential facts schemas. The natural language task description is subsequently converted into a planning goal. Following the initial exploration phase, the semantic spatial graph is updated to indicate the new position of the agent and its perceptions. The algorithm iterates through these steps: a) finding a plan for achieving the goal and returning it upon success; b) if no plan exists, replacing the goal with another fact, called (explore), associated with unvisited states. The semantic spatial graph is updated during each iteration to construct the new initial state of the agent, allowing for an incremental egocentric view of the environment.
4 Embodied Agents: Generalization within Object Types and Skills
Embodied Agents are task solvers in environments linked to a hidden Object-oriented POMDP, emphasizing objects, actions/skills, and costs rather than rewards. We assume that these objects mediate perceptions and actions. An Environment is a tuple,  $\mathcal{E} = \langle A_{\mathcal{E}},\mathcal{T}_{\mathcal{E}},\mathcal{V}_{\mathcal{E}},\text{reset},\text{step}\rangle$ , with  $A_{\mathcal{E}}$  being the set of parameterized actions,  $\mathcal{T}_{\mathcal{E}}$  defining the set of object types and potential values  $\mathcal{V}_{\mathcal{E}}$ , reset representing the first observation, and step representing an action execution that returns an observation and cost. A Task is represented by a tuple,  $T_{\mathcal{E}} = \langle I_{\mathcal{E}},G_{\mathcal{E}}\rangle$ , which sets the initial state and the goal of the environment via the reset function. Lastly, Embodied Agents employ functions  $M_I$  to initialize their mental state and  $M_U$  to update it, have a policy  $\pi_M$  to select actions and a function  $G_M$  for deciding when they consider the goal to be achieved.
Agents that tackle a stream of tasks within the same environment have incentives to specialize to the known object types and their inherent capabilities expressed through parameterized actions. Such over-specialization provides an opportunity for developing robust agents that can generalize across different tasks within the same environment. This principle applies to physical and software agents whose actions are grounded in these objects. Our framework forms a basis for creating flexible and adaptable agents that can perceive and act upon recognized object types. Further details and implications are covered in Section D.
5 Egocentric Planning for Embodied Agents
Here, we introduce Egocentric Planning, an approach for embodied agents to solve tasks in Object-oriented POMDP environments. The approach requires a symbolic planning model that assumes full observability for testing both action applicability and goal achievement. However, the complexity of specifying or learning such a symbolic model is simplified by its alignment with the Object-oriented POMDP environment, object types, and action signatures. Egocentric planning relies on a reasonable assumption to derive a sensor model, equivalent to a function that updates the distribution of possible
current states given new observations (APG09). For instance, in a planning domain with objects for different types and colors, the sensor model can say that cubes and rectangles are red, while ovals and circles are black. When the agent examines an object, and observes that it is black, the agent should filter the belief to represent that the object cannot be a cube or a rectangle. In contrast, our method requires a set of anchor object types that are assumed to reveal information about other objects and exploration actions that allow visiting new anchor objects, revealing new objects and their properties. The method can be combined with standalone models for processing images or natural language. The resulting method is both theoretically sound, assuming that the symbolic planning model is correct, and practically applicable, as it leverages the speed of symbolic planners.
We present an abstract algorithm that leverages the objects types and action signatures of the Object-oriented POMDP to ground the required symbolic planning model.
5.1 Background: Parameterized Full-Observable Symbolic Planning
In the supplementary material, Section E, we define Parameterized Full-Observed Symbolic Planning domains and problems. A planning domain is defined as a tuple  $\mathcal{PD} = \langle \mathcal{T},\mathcal{V},\mathcal{P},\mathcal{A},\Phi \rangle$  where  $\mathcal{T}$  is a finite set of object types with  $\mathcal{V}$  a finite set of values,  $\mathcal{P}$  is a finite set of typed predicates,  $\mathcal{A}$  is a finite set of typed actions with preconditions and effects, and  $\Phi$  is a function that maps each action and predicates to a tuple expressing their typed arguments. A planning problem consists of a planning domain, a finite set of objects, and descriptions of the initial state and the goal.
For ALFRED, we rely on Classical Planning, a kind of full-observable planning that assumes deterministic actions, where preconditions are a set of predicates, and effects are a pair of sets of predicates called add and delete, representing the action's modifications. Correspondingly, a parametric classical planning problem is a full-observable planning problem where the initial state and the goal are conjunctions of grounded predicates.
5.2 Egocentric Planning for Embodied Agents
Egocentric planning employs a user-specified fully-observable symbolic planning model to address tasks in Object-oriented POMDP environments. In the next section, we detail the specifics of our ALFRED implementation. As actions in ALFRED are deterministic, we can apply a classical planner to solve the associated planning problems. We first present a general algorithm for parametric full-observable planning problems.
Algorithm 1  
Iterative Exploration Replanning (IER)  
Input: Environment  $\langle A_{\mathcal{E}},\mathcal{T}_{\mathcal{E}},\mathcal{V}_{\mathcal{E}}$  reset,step>  
Input: Planning domain  $\mathcal{PD} = \langle \mathcal{T},\mathcal{V},\mathcal{P},\mathcal{A},\Phi \rangle$   
Input: Anchor Types & Exploration Acts  $\langle \mathcal{T}_a,\mathcal{X}\rangle$   
Input: Mental state Init & Update  $\langle M_I^{PD},M_U^{PD}\rangle$   
Input: Task  $\langle I_{\mathcal{E}},G_{\mathcal{E}}\rangle$   
Output: Successful trace  $\tau$  or Failure  
1:  $p\gets$  reset  $(I_{\mathcal{E}},G_{\mathcal{E}})$  ▷ Initial perception  
2:  $\mathcal{O},I,G\gets M_I^{\mathcal{PD}}(p,I_{\mathcal{E}},G_{\mathcal{E}})$   
3:  $\mathcal{C}\gets \{o^{\prime}\mid \mathrm{type}(o^{\prime})\in \mathcal{T}_{a}$  and  $o^{\prime}$  occurs in  $I\}$  ▷ C: Observed Anchor Objects  
4:  $\tau$  solved  $\leftarrow []$  False  
5: while not solved do  
6:  $\pi_{solve}\gets$  Solve  $(\langle \mathcal{PD},\mathcal{O},I,G\rangle)$   
7: if  $\pi_{solve}$  is None then  
8:  $\mathcal{A}_e\gets \mathcal{A}$   
9: for  $o\in \mathcal{O}$  and type  $(o)\in \mathcal{T}_a$  do  
10: if  $o\notin \mathcal{C}$  then  
11:  $I\gets I\cup \{(unknown o)\}$   
12: if (unknown  $o)\in I$  then  
13:  $\mathcal{A}_e\gets \mathcal{A}_e\cup \{\text{ExploreAct}(a,o)$  for  $a\in \mathcal{X}\}$   
14:  $G_e\gets \{(\text{explored})\}$   
15:  $\pi_{explore}\gets$  Solve  $(\langle \mathcal{PD}\text{with}\mathcal{A}_e,\mathcal{O},I,G_e\rangle)$   
16: return Failure if if  $\pi_{explore}$  is None  
17: for  $a\in \pi_{explore}$  do  
18:  $p,c\gets$  step(a)  
19: Break if failed  $\in p$   
20:  $\tau .$  append(a)  
21:  $\mathcal{O},I\gets M_U^{\mathcal{PD}}(\mathcal{O},I,a,p,c)$   
22: if  $a\in X$  then  
23:  $\mathcal{C}\gets \mathcal{C}\cup \{o^{\prime}\mid \mathrm{type}(o^{\prime})\in \mathcal{T}_a$  and  $o^{\prime}$  argument of  $a\}$   
24: else  
25: for  $a\in \pi_{solve}$  do  
26:  $p,c\gets$  step(a)  
27: Break if failed  $\in p$   
28:  $\tau .$  append(a)  
29:  $\mathcal{O},I\gets M_U^{\mathcal{PD}}(\mathcal{O},I,a,p,c)$   
30: solved  $\leftarrow$  True if  $I$  satisfies  $G$   
31: return  $\tau$
Algorithm 1 implements our Egocentric Planning approach by constructing a sequence of planning problems, alternating exploration and task solving. At each step, the agent's mental state is updated though sensory input to represent the current state and the known objects that will be part of the
following planning problem. The agent deems the goal achieved when it finds a plan for achieving it and executes the plan successfully. Our policy  $(\pi_{M})$  leverages a standalone planner for both exploration and task solving.
Our exploration and sensing strategy hinges on two elements. First, anchor object types  $\mathcal{T}_a \subseteq \mathcal{T}$  are types of objects that reveal the presence of others. For example, in ALFRED, location is the sole anchor object type. Second, exploration actions  $\mathcal{X} \subseteq \mathcal{A}$  enables the discovery of previously unknown objects. For instance, moving to a new location in ALFRED enables observing new items and locations. This object-centric view of the environment simplifies the creation of a symbolic model required for planning. The function ExploreAct  $(a, o)$  returns a copy of the function where (unknown  $o$ ) is added to the precondition, and the effect is extended to cause (explored) to be true and (unknown  $o$ ) to be false.
Our agent's mental state is represented by a set of objects and an initial state of grounded predicates,  $(\mathcal{O},I)$ . The function  $M_I^{\mathcal{P}\mathcal{D}}$  initializes the symbolic goal  $G$  and sets  $(\mathcal{O},I)$  with the initial objects and true grounded predicates. The mental state update function,  $M_U^{\mathcal{P}\mathcal{D}}$ , updates  $(\mathcal{O},I)$  given the executed action  $a$ , the returned perception  $p$ , and the cost  $c$  of executing  $a$ . For example, the mental state update in ALFRED adds new objects observed in the perception. It updates the initial state with these new objects and their properties, including their relationship with non-perceived objects like location. Both functions maintain the alignment between the current planning state and the environment with an underlying object-oriented POMDP model.
Specifically, the actions, object types, and the values of the environment may be different from the planning domain's. While the ALFRED environment supports movement actions like move-forward or turn-left, the planning domain might abstract these into a move action. Although the ALFRED environment does not include a location type, we manually specify it so movable objects can be associated with a specific location. Indeed, the first initial state  $I$  includes a first location and its surrounding ones. Thus, the agent can revisit the location once it has found a solution involving that object. While integrating these components may be challenging, it is a hurdle common to all embodied agent approaches aiming to generalize across components under a constrained training budget. In general, practical applications will likely monitor the agent's performance regarding object types and skills. We discuss the advantages and drawbacks of our approach in Section 9.
6 Approach for ALFRED
6.1 Model Overview
Our method is structurally similar to FILM but we substitutes the SLAM-style map with a graph structure for object and location information storage  $(\mathrm{MCR}^{+}22)$ . The action controller employs our iterative egocentric planner that utilizes a semantic location graph for task completion and exploration. The node of the graph are uniquely labeled by the coordinates (x-coordinate, y-coordinate, facing direction), and the edges are labeled by actions (turn-right, turn-left, move-forward). Each node in the graph stores object classes, segmentation masks, and depth information generated by the vision module. We label the nodes with "known" if the agent has visited the particular location and "unknown" if the node is only observed. Unknown nodes on the graph are prioritized for exploration based on objects they contain. This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks. At each timestep, the vision module processes an egocentric image of the environment into a depth map using a U-Net, and object masks using Mask2-RCNN (HGDG17,  $\mathrm{LCQ}^{+}18$ ). The module then computes the average depth of each object and stores only those with an average reachable distance less than 1.5. A confidence score, calculated using the sum of the object masks, assists the egocentric planner in prioritizing object interaction. We employ text classifier to transform high-level language task description into goal conditions. Two separated models determine the task type and the objects and their properties. For instance, a task description like "put an apple on the kitchen table" would result in the identification of a "pick-up-and-place" task and a goal condition of (on apple table). We convert the FILM-provided template-based result into goal conditions suitable for our planner. The spatial graph, acting as the agent's memory during exploration, bridges grounded objects and the state representation required for our planner. The graph, updated and expanded by policies produced by our egocentric planner, encodes location as the node key and visual observations as values, with edges representing agent actions.
6.2 Egocentric Agent for ALFRED
The object-oriented environment for ALFRED, denoted as  $\mathcal{E} = \langle A_{\mathcal{E}},\mathcal{T}_{\mathcal{E}},\mathcal{V}_{\mathcal{E}},\text{reset},\text{step}\rangle$ , includes actions such as pick-up, put-down, toggle, slice, move-forward, turn-left, and turn-right. The FILM vision module is trained as flat recognizer of all possible object types. We postprocess such detection into high level types Object and Receptacle in  $\mathcal{T}_{\mathcal{E}}$ , with corresponding  $\mathcal{V}_{\mathcal{E}}$  indicating subtypes and additional properties. For instance, the object type includes subtypes like apple and bread, while Receptacle includes subtypes such as counter-top and microwave. Values incorporate properties like canHeat, canSlice, isHeated, and isCooled.
The planning domain for ALFRED is denoted  $\mathcal{PD} = \langle \mathcal{T},\mathcal{V},\mathcal{P},\mathcal{A}\rangle$ .  $\mathcal{T}$  includes the Location type, with corresponding values in  $\mathcal{V}$  representing inferred locations as actions are deterministic. Predicates in  $\mathcal{P}$  represent the connection between locations and the presence of objects at those locations. Further predicates express object properties, such as canHeat, canSlice, isHeated, and isCooled. Unlike the ALFRED environment, actions in the planning domain are parametric and operate on objects and locations. Instead of move-forward, the planning domain features a move action, representing an appropriate combination of move-forward, turn-left, and turn-right. For actions like toggle in ALFRED, which have different effects depending on the object in focus, we introduce high-level, parametric actions like heat that can be instantiated with the relevant object. As the actions in ALFRED are deterministic, we use a classical planner, enabling high scalability by solving one or two fresh planning problems per iteration of Algorithm 1.