SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, with limited resources or data, generalizing to new situations can be challenging. Take an embodied agent in an environment (def D.8) with  $N$  skills and  $M$  types of objects, tasked with carrying out a sequence of tasks that require plans involving a new combination of  $k$  skills and  $m$  types of objects, where  $n \ll N$  and  $m \ll M^3$ . Such setting is within the scope of methods like meta-learning or continual learning for adapting the latent representation for such unfamiliar environments (FAL17). The priors in some modular meta-learning approaches are related with our notions of objects and actions, but their inference algorithms might not scale well for long plans or ignore that some users might also want intentional modelling (ALPK18, ORCC21, MvSE22). Egocentric planning, on the other hand, may provide predictable scalability in these scenarios. While the definition of open-world generalization might be more elusive, our method leans towards systematic generalization across actions and object types.
Planning using certain latent representations or world models can also prove costly. As explained by (AKFM22) (LATPLAN), not all factored representations afford the same level of scalability. The authors of LATPLAN apply variational autoencoders to learn a discrete latent representation, incorporating additional priors so the representation can be interpreted as a STRIPS symbolic model, the most basic type of symbolic model we use for egocentric planning. Symbolic models like STRIPS enable solvers with greater scalability and generalization.
An alternative approach involves methods that learn a symbolic planning model  $(\mathrm{AFP}^{+}18, \mathrm{CDVA}^{+}22)$ . Object-oriented POMDPs are especially suitable as the grounding on objects and action labels allows for more efficient algorithms (CMW13, MCRW09), an advantage not enjoyed by the authors of LATPLAN as they only use pairs of images as training data. Learning a symbolic model helps manage the risk of compounding errors since the model can be debugged and improved. Real-world applications may require monitoring the agent's behaviour for specific objects or actions. Furthermore, specific actions may carry inherent risks that need to be avoided. In general, we anticipate that developers of embodied agents will use as much learning as possible, as much intentional modelling as necessary, and symbols to blend the two of them  $(\mathrm{KSV}^{+}22)$ .
Further integration between symbolic planning models and other ML models might help to reduce some limitations of our approach. For instance, in ALFRED, searching for a pan should make the agent head toward the kitchen. If another model like an LLM finds it likely that the pan is in the kitchen, we can use the scores to produce action cost in a similar way to the integration done in SayCan,  $(\mathrm{iBC}^{+}23)$ .
Egocentric planning connects to a body of work aimed at tackling expressive planning problems by harnessing the scalability of classical planners. While egocentric planning focuses on the partial observability of objects, (APG09, PG09) consider partial observability given a set of possible initial
3 While the state space is unknown to the agent, it is aware of the actions and object types.
states and deterministic actions, and (MMB14) considers full observability but with nondeterministic actions. Though these strategies can provide complete solutions proven offline, they can also serve as approximate methods for solving online problems. When planning with probabilistic effects, replanning assuming randomized determinization of actions can be a strong approach if there are no dead ends (YFG07).
Our symbolic planning models are extensions of STRIPS, but other options exist. Hierarchical task networks (HTN)  $(\mathrm{NAI}^{+}03,\mathrm{GA14})$  , for example, offers an alternative to STRIPS as they allow the specification of plans at various abstraction levels. Sometimes, it is more practical to outline potential plans using formalisms like linear temporal logic (LTL) formulae (DV13). Nonetheless, the relationship between these models has been examined, and it is feasible to integrate them with STRIPS-based planning (CC04,  $\mathrm{HBB}^{+}19$  , MH13).
C Results in ALFRED's Task and New Tasks
C.1 ALFRED Task Performance
ALFRED Tasks	Valid Unseen	Valid Seen
SR	GC	SR	GC
look_at_obj_in_light	53.77	54.27	47.22	30.11
pick_and_placesimple	43.14	47.11	37.15	39.77
pick_coolthen_place_in_recep	46.76	45.82	47.01	48.89
pick.cleanthen_place_in_recep	38.77	41.37	38.01	42.21
pick_coolthen_place_in_recep	46.26	46.81	47.21	47.33
pick_heatthen_place_in_recep	37.69	40.50	39.82	41.33
pick_two_obj_and_place	30.33	35.71	27.37	31.21
To analyze the strengths and weaknesses of our approach in different types of tasks, we conducted further evaluations by task type as in Table 5. Our findings reveal several noteworthy observations. Firstly, the success rates (SR) and goal completion rates (GC) for the "Pick_two_obj_and_place" task are lower than the rest, mostly due to these tasks involving more objects and with smaller masks. The task "look_at_obj_in_light" has the best performance due to the variety of lamp is low compare to other common objects. Secondly, there is no strong correlation between the number of interactions involved in a task type and its performance  $(\mathrm{STG}^{+}20)$ . For example, the task "pick_cool_then_place_in_receptacle" has an extra cooling step compared to "pick_and_placesimple" but still have better performance. A detailed description of each task can be found in the original ALFRED paper on page 16 to 19  $(\mathrm{STG}^{+}20)$ .
C.2 New Task Performance
In section 8.4, we discuss the generalization of our approach to new tasks. The results are summarized in table 6.
Table 5: Performance on Valid Unseen and Valid Seen for each task
SR	GC	PLWSR	PLWGC
Pickup and Place Two Objects	90.00	93.44	3.71	4.37
Clean-and-Heat Objects	80.00	89.77	2.56	3.90
Clean-and-Cool Objects	85.00	93.38	4.44	4.78
Heat-and-Cool Objects	80.00	89.20	4.01	4.66
Pick and Place Object in Printer	70.00	77.81	2.59	3.22
Table 6: New tasks
Our approach enables us to reuse the definition of existing models to achieve zero-shot generalization to new tasks as long as the set of actions, objects and their relationships remain the same. In many embodied agent applications, agent need to have ability to adapt to new tasks. For instance, organizing a living room could involve storing loose items on a shelf or arranging them on a table. An
agent familiar with common household objects and equipped with the skills to handle them should seamlessly manage both types of cleaning tasks. However, transfer learning for neural networks remains challenging in the context of embodied agent tasks generalizing across objects and skills. Similarly, the template system used in FILM demands hand-crafted policies for new task types. The ALFRED dataset include instructions for each tasks that  $(\mathrm{STG}^{+}20)$ , although our approach does not use them. In a our egocentric planning formulation, a task can be broken down into a set of goal conditions that comprise objects of interest. The planner can then autonomously generate an action sequence to reach these goals. Consequently, our method can independently execute these tasks without the need for transfer learning or the addition of new task templates. To demonstrate this, we devised 5 new task types that were not present in the training data. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made to minimize perception errors. These tasks share the same world model as ALFRED and thus do not necessitate any additional modification beyond specifying a new goal. We selected environments and layouts from the training set where subgoals required for each new task type are feasible, ensuring the feasibility of the new task for our agent. Table 6 lists these new tasks and their success rates. We selected 20 scenarios for each new task and maintained the maximum 10 interaction error as in the ALFRED problem. Our method was able to achieve an impressive overall success rate of  $82\%$  without any additional adjustments beyond the new goal specification. However, due to the constraints of the ALFRED simulator, we're unable to experiment with introducing new objects and action types beyond what's available. For instance, "heating" in ALFRED is solely accomplished via a microwave. In real-world scenarios, we could easily incorporate ovens and actions to operate them with minor alterations to our PDDL domains. Similarly, we could introduce time and resource restrictions for each action, which are common in many PDDL benchmarks. The ability to accommodate new goals and constraints is a key advantage of our planning-based approach.
C.3 Computation
Perception and Language module was fine-tuned on a Nvidia 3080. The planning algorithm uses a 8 core Intel i7 CPU with  $32\mathrm{GB}$  of ram. More detailed on training can be found in  $(\mathrm{MCR}^{+}22)$
D Embodied Agents Generalizing within given Objects Types
In this section, we provide further details about embodied agents presented in section 4. Embodied agents perceive and act in environments to accomplish tasks. Navigating challenges and constraints imposed by their environment and tasks, these agents ensure safety and adhere to physical restrictions. Taskable agents rely on their surroundings and inherent capabilities, creating a strong dependency and often specializing in particular environments. This specialization shapes their cognition and problem-solving strategies, presenting an opportunity for robust agent design.
We exploit this limitation by focusing on generalization across tasks involving the same object types and their relationships, applicable to both physical and software agents. Taskable agents possess specialized capabilities for these object types, as seen in robotics where battery level, body state, and actuators are distinct variables  $(\mathrm{CST}^{+}22)$ . Object types also serve as inputs/outputs in end-to-end approaches with learned representations and appear in embodied agents operating in software environments, using APIs and maintaining internal states. Our symbolic planning approach aims to provide a basis for designing adaptable and flexible agents that sense and act upon given object types. We start by defining object types, and entities grounded in these types.
Definition D.1 (Objects and Object Types). An Object Type, denoted as  $\langle \mathcal{T},\mathcal{V}\rangle$  , consist of a finite set of object types  $\mathcal{T}$  along with a mapping  $\mathcal{V}:t_i\to \mathcal{V}_i$  that assigns each type  $t_i\in \mathcal{T}$  a set of possible values  $\mathcal{V}_i$  . An Object  $o = (t_i,v)$  is an instance of a type with a specific value, i.e.,  $t_i\in \mathcal{T}$  and  $v\in \mathcal{V}(t_i)$
Example D.2 (Object types and Values). An example of an object type is  $t_{objectSeen}$  with values  $\mathcal{V}_{(x,y)} \times \mathcal{V}_{\text{direction}} \times \mathcal{V}_{\text{objectSubtype}} \times \mathcal{V}_{id}$  where  $\mathcal{V}_{(x,y)}$  is position in a grid,  $\mathcal{V}_{\text{direction}}$  is the direction the object is facing,  $\mathcal{V}_{\text{objectSubtype}}$  is the category of the object observed, and  $\mathcal{V}_{id}$  is a unique identifier for the object. Other related examples of object types are  $t_{movable}$  with values  $\mathcal{V}_{movableSubtype} \times \mathcal{V}_{id} \times \mathcal{V}_{\text{objectproperties}}$ , and  $t_{obstacle}$  with values  $\mathcal{V}_{\text{obstacleSubtype}} \times \mathcal{V}_{id} \times \mathcal{V}_{\text{objectproperties}}$ . Furthermore, an action pickup with parameters  $\Phi_{\mathcal{E}}(a) = (t_{\text{objectSeen}})$  is an action that picks up a specific object.
Object types are used as parameters of entities of actions of Object-oriented POMDP defined, and as parameters of actions and predicates in planning domains in section E.
Definition D.3 (Typed Entity). Given a set of entities  $\Xi$ , object types  $\langle \mathcal{T},\mathcal{V}\rangle$ , and a mapping  $\Phi :\xi \to \bigcup_{k\in \mathbb{N}}\mathcal{T}^k$ . A Type Entity  $\xi \in \Xi$  is mapped to  $\Phi (\xi) = (t_1,t_2,\dots ,t_{k_\xi})$ , with each  $t_i\in \mathcal{T}$ . This mapping associates each entity with a  $k_{\xi}$ -tuple of object types, corresponding to  $k_{\xi}$  parameters of the entity, defining the arity of the entity  $\xi$ . We refer to the variables  $var\in \{var_1, var_2,\dots, var_{k_\xi}\}$  as the positional parameters of the entity  $k_{\xi}$ , denoted  $\xi(var_1, var_2, \dots, var_k)$ .
Definition D.4 (Grounded Entity). Given a set of typed entities  $\Xi$ , with object types  $\langle \mathcal{T}, \mathcal{V} \rangle$ , and a mapping  $\Phi$ , a Grounded Entity, denoted as  $\xi^g$  for  $\xi \in \Xi$ , is a parametric object where each type argument of  $\xi$  specified by  $\Phi(\xi)$  is replaced by an object of the corresponding type. Specifically,  $\xi^g = \xi(o_1, o_2, \ldots, o_{k_\xi})$ , where each  $o_i = (t_i, v_i)$  with  $v_i \in \mathcal{V}(t_i)$ . The set of all grounded entities is denoted by  $\Xi^g$ .
Now we define the environment where the embodied agent operates as a Partially Observable Markov Decision Process (POMDP) with costs. Cost POMDPs are related to classical MDPs but feature partial observability and use costs instead of rewards (MK12, GB13).
The literature sometimes confound the environment with the underlying the POMDP, but we want to make explicit what is available to the agent. Thus, while our underlying model is a POMDP, we consider agents that can aim to generalize without being fully aware of the distribution of future tasks, except for the actions and object types that are explicit in the environment. Therefore, we abuse slightly the notation to annotate with the objects defined below with the environment, denoted  $\mathcal{E}$ .
Definition D.5 (Cost POMDP). A Cost POMDP is a tuple  $\langle S_{\mathcal{E}},A_{\mathcal{E}},T_{\mathcal{E}},C_{\mathcal{E}},P_{\mathcal{E}},Z_{\mathcal{E}}\rangle$  , where  $S_{\mathcal{E}}$  is a set of states,  $A_{\mathcal{E}}$  is a finite set of actions,  $T_{\mathcal{E}}:S_{\mathcal{E}}\times A_{\mathcal{E}}\times S_{\mathcal{E}}\to [0,1]$  is the transition function,  $C_\varepsilon :S_\varepsilon \times A_\varepsilon \to \mathbb{R} > 0$  is the cost function,  $P_{\mathcal{E}}$  is a set of perceptions, and  $Z_{\mathcal{E}}:S_{\mathcal{E}}\times A_{\mathcal{E}}\times P_{\mathcal{E}}\times S_{\mathcal{E}}\rightarrow$ $[0,1]$  is the sensor function.
Object-oriented POMDP is an extension of a Cost POMDP that includes objects and their properties to be used for expressing perceptions and actions.
Definition D.6 (Object-oriented POMDP). An Object-oriented POMDP  $\Gamma$ , represented by  $\langle S_{\mathcal{E}}, A_{\mathcal{E}}, \Phi_{\mathcal{E}}, T_{\mathcal{E}}, C_{\mathcal{E}}, P_{\mathcal{E}}, Z_{\mathcal{E}}, T_{\mathcal{E}}, \mathcal{V}_{\mathcal{E}} \rangle$ , extends a Cost POMDP by incorporating a predefined set of Object Types  $\langle \mathcal{T}_{\mathcal{E}}, \mathcal{V}_{\mathcal{E}} \rangle$ . It defines the set of perceptions  $P_{\mathcal{E}}$  where a perception  $p \in P_{\mathcal{E}}$  is a finite set of objects  $\{o_1, o_2, \ldots, o_m\}$ , and the set of typed actions  $A_{\mathcal{E}}$  defined for the object types  $\langle \mathcal{T}_{\mathcal{E}}, \mathcal{V}_{\mathcal{E}} \rangle$  and the map  $\Phi_{\mathcal{E}}$ .
In this context, Grounded Actions refer to grounded entities, where the set of entities is the actions  $A_{\varepsilon}$ . Values in object-oriented POMDPs can be dense signals like images or sparse signals like object properties. Values can have any structure, so they can refer to values in the domain of other types. Indeed, Object-oriented POMDPs can express Relational POMDPs (WK10, WJK08, SB09, TGD04, WPY05, DNR+20). However, our approach emphasizes the compositionality of the objects and their properties.
Agents interact with the POMDP through an environment. Given a task, we initialize the environment using the task details. The agent receives the tasks and acts until it achieves the goal or fails due to some constraint on the environment. The agent achieves a goal if, after a sequence of actions, the environment's hidden state satisfies the goal.
Definition D.7 (Task). Given an Object-oriented POMDP  $\Gamma$ , a Task is a tuple  $T_{\mathcal{E}} = \langle I_{\mathcal{E}}, G_{\mathcal{E}} \rangle$ , where  $I_{\mathcal{E}}$  is an object that the environment interpret for setting the initial state, and  $G_{\mathcal{E}}$  is a set of goal conditions expressed as objects  $\{g_1, g_2, \ldots, g_n\}$ , with each  $g_i$  being an instance of an object.
Definition D.8 (Object-oriented Environment). Given a hidden Object-oriented POMDP  $\Gamma$ , an Object-oriented Environment, or just Environment, is a tuple  $\mathcal{E} = \langle A_{\mathcal{E}}, \mathcal{T}_{\mathcal{E}}, \mathcal{V}_{\mathcal{E}}, \text{reset}, \text{step} \rangle$ , where  $A_{\mathcal{E}}, \mathcal{T}_{\mathcal{E}}$  and  $\mathcal{V}_{\mathcal{E}}$  are as previously defined, reset:  $T_{\mathcal{E}} \to P_{\mathcal{E}}$  is a function that takes a task  $T_{\mathcal{E}} = \langle I_{\mathcal{E}}, G_{\mathcal{E}} \rangle$ , resets the environment's internal state using  $I_{\mathcal{E}}$  and returns an initial observation, and step:  $A_{\mathcal{E}}^{g} \to P_{\mathcal{E}} \times \mathbb{R}$  is the interaction function that updates the environment's internal state according to the execution of a grounded action  $a_{g}$ , for  $a \in A_{\mathcal{E}}$ , and returns an observation and a cost.
In some environments,  $I_{\mathcal{E}}$  might be opaque for the agent. In others, it might containing information not used by the environment like user's preferences about how to complete a task. However, we abuse this notion and refer to  $I_{\mathcal{E}}$  as the initial state of the environment.
Separating Object-oriented POMDPs from environments allows us to consider the scope of the generalization of our agents within the same POMDP. While the agent cannot see the state space directly, it is aware of the actions and the object types.
Definition D.9 (Embodied Agent). Given an environment  $\mathcal{E} = \langle A_{\mathcal{E}},\mathcal{T}_{\mathcal{E}},\mathcal{V}_{\mathcal{E}},\text{reset, step}\rangle$ , and a task  $T_{\mathcal{E}} = \langle I_{\mathcal{E}},G_{\mathcal{E}}\rangle$ , an Embodied Agent is a tuple  $\langle M_I,\pi_M,M_U,G_M\rangle$ , where  $M_I:P_{\mathcal{E}}\times I_{\mathcal{E}}\times G_{\mathcal{E}}\to \mathcal{M}$  is a function that takes an initial observation, a task, and returns the initial internal state of the agent,  $\pi_M:\mathcal{M}\to A_\varepsilon^g$  is the agent's policy function that maps mental states to a grounded action in  $A_{\varepsilon}^{g}$ ,  $M_U:\mathcal{M}\times A_\varepsilon^g\times P_\varepsilon \times \mathbb{R}\to \mathcal{M}$  is the mental state update function receiving a grounded executed action, the resulting observation, and the cost, and  $G_{M}:\mathcal{M}\rightarrow [0,1]$  is the goal achievement belief function that returns the belief that the agent has achieved the goal based on its mental state  $M$ .
Algorithm 2 provides a general framework for an agent's execution process. By instantiating the policy, mental state update function, and goal achievement belief function using appropriate techniques, the agent can adapt to a wide range of environments and tasks. Although new tasks can set the agent in unseen regions of the state space  $S_{\varepsilon}$ , the agent remains grounded on the known types and the actions parameterized on object types,  $\mathcal{T}_{\varepsilon}$ . They offer opportunities for generalization by narrowing down the space of possible policies and enabling compositionality. The agent's mental state  $M$  can include its belief state, internal knowledge, or other structures used to make decisions. The policy  $\pi_{M}$  can be a deterministic or stochastic function, learned from data or user-specified, or a combination of both. As the agent has an internal state, its policy can rely on a compact state or on the history of interactions.
Algorithm 2 Agent Execution
Input: Environment  $\langle A_{\varepsilon},\mathcal{T}_{\varepsilon},\mathcal{V}_{\varepsilon},\mathrm{reset,step}\rangle$
Input: Agent  $\langle M_I, \pi_M, M_U, G_M \rangle$
Input: Task  $\langle I_{\mathcal{E}},G_{\mathcal{E}}\rangle$
Input: Probability threshold  $\theta$
Output: Successful trace  $\tau$  or Failure
1:  $p \gets \text{reset}(I_{\mathcal{E}}, G_{\mathcal{E}})$
2:  $\tau = []$ $\triangleright$  Empty trace
3:  $M\gets M_I(p,I_{\mathcal{E}},G_{\mathcal{E}})$
4: while  $G_{M}(M) < \theta$  do
5:  $a^g = a(o_1,o_2,\dots,o_{k_a})\sim \pi_M(M,A_\mathcal{E}^g)$
where  $a$  in  $A_{\varepsilon}$  and  $o_i$  are objects.
6:  $\tau$ .append  $(a^g)$
7:  $p^{\prime},c\gets \mathsf{step}(a^{g})$
8: if failed  $\in p^{\prime}$  then
9: return Failure
10:  $M\gets M_U(M,a^g,p',c)$
11: return  $\tau$
The agent starts in the initial state set from  $I_{\mathcal{E}}$  and aims to achieve the goal  $G_{\mathcal{E}}$ . While the agent does not know the true state of the environment, agents can aim to generalize across actions and the object types as they are always available. While we prefer policies with lower expected cost, our main concern is achieving the goal. It is possible that a task might be unsolvable, possibly because of actions taken by the agent. We assume that failures can be detected, for instance, by receiving a special observation type failed. We further assume that after failure, costs are infinite, and failed is always observed.
The Iterative Exploration Replanning (IER), Algorithm 1 in section 5, is a instance of algorithm 2. Instead of using a parameter  $\theta$ , it considers the problem solved when it obtains a plan that achieves the goal.
E Detailed Background: Parameterized Full-Observable Symbolic Planning
In this section, we describe the background definitions of parameterized full-observable symbolic models mentioned in section 5.1. We define planning domains and problems assuming full observability, including classical planning that further assumes deterministic actions, also known as STRIPS (GNT04, GB13). Planning domains and problems rely on typed objects (Def D.1), as object-oriented POMDPs, but planning actions include a model of the preconditions and effects.
Definition E.1 (Parametric Full-Observed Planning Domain). A parametric full-observable planning domain is a tuple  $\mathcal{PD} = \langle \mathcal{T},\mathcal{V},\mathcal{P},\mathcal{A},\Phi \rangle$ , where  $\langle \mathcal{T},\mathcal{V}\rangle$  is a set of object types,  $\mathcal{P}$  is a set of typed predicates and  $\mathcal{A}$  is a set of typed actions, both typed by the object types and the map  $\Phi$ . Each action  $a$  has an associated  $\mathrm{PRE}_a$ , expressing the preconditions of the action, and  $\mathrm{EFF}_a$ , expressing its effects. For each grounded action  $a(o_1,o_2,\dots,o_k)\in A^g$ , the applicability of the grounded action in a state  $s$  is determined by checking whether the precondition  $\mathrm{PRE}_a$  is satisfied in  $s$ . The resulting state after applying the grounded action is obtained by updating  $s$  according to the effect update  $\mathrm{EFF}_a$ . The specification of  $\mathrm{EFF}_a$  is described below.
While the values in both parametric full-observable planning domains and object-oriented POMDPs can be infinite (def D.6), the values used in concrete planning problems are assumed to be finite. As a consequence, the set of grounded actions of a planning problem is finite (def D.4).