SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition E.2 (Parametric Full-Observed Planning Problem). A parametric full-observable planning problem is a tuple  $\langle \mathcal{PD},\mathcal{O},I,G\rangle$ , where  $\mathcal{PD} = \langle \mathcal{T},\mathcal{V},\mathcal{P},\mathcal{A},\Phi \rangle$  is a parametric full-observable planning domain,  $\mathcal{O}$  is a finite set of objects, each associated with a type in  $\mathcal{T}$ ,  $I$  is a description of the initial state specifying the truth value of grounded predicates in  $\mathcal{P}^g$ ,  $G$  is a description of the goal specifying the desired truth values of grounded predicates in  $\mathcal{P}^g$ , where  $\mathcal{P}^g$  are the predicates  $\mathcal{P}$  grounded on object types  $\langle \mathcal{T},\mathcal{V}\rangle$  and the map  $\Phi$  encoding their typed arguments.
This definition covers a wide range of symbolic planning problems featuring full observability including probabilistic effects (KHW95). In particular, the definition can be specialized for classical planning problems, where the actions are deterministic, corresponding to STRIPS, to the most studied symbolic model (GB13).
For classical planning, given an action  $a \in \mathcal{A}$  parameterized by a list of object types  $a(t_1, t_2, \ldots, t_k)$ , their precondition or effect are formulas  $\gamma$  express in terms of the variables  $var \in \{var_1, var_2, \ldots, var_k\}$  that refer to positional parameters of the action. We denote the grounded formula  $\gamma(o_1, o_2, \ldots, o_k)$  as  $\gamma$  with each variable  $var_i$  that occurs in  $\gamma$  is replaced by corresponding objects  $o_i$ .
Definition E.3 (Parametric Classical Planning Domain). A parametric classical planning domain is a parametric full-observable planning domain where each action  $a \in \mathcal{A}$  has preconditions  $\mathrm{PRE}_a$ , and an effect update  $\mathrm{EFF}_a$  represented as a tuple  $(\mathrm{ADD}_a, \mathrm{DEL}_a)$  of add and delete effects, all sets of predicates in  $\mathcal{P}$ . Given a grounded action  $a(o_{arg})$  with  $o_{arg} = o_1, o_2, \ldots, o_k$ , the precondition  $\mathrm{PRE}_a$  is satisfied in a state  $s$  if  $\mathrm{PRE}_a(o_{arg}) \subseteq s$ . The effect update  $\mathrm{EFF}_a$  describes the resulting state as  $(s \setminus \mathrm{DEL}_a(o_{arg})) \cup \mathrm{ADD}_a(o_{arg})$  after applying the action.
Definition E.4 (Parametric Classical Planning Problem). Given a parametric classical planning domain  $\mathcal{PD}$ , a parametric classical planning problem is a parametric full-observable planning problem where  $I$  and  $G$  are conjunctions of predicates in  $\mathcal{P}$  (equivalently viewed as sets).
As Egocentric Planning relies on replanning over a sequence of related planning problems, it is convenient to define some operations required for exploration, as the agent discovers more objects. They allow us to implement the function ExploreAct in Algorithm 1.
Definition E.5 (Precondition Extension). Given a parametric full-observable planning problem and an action  $a \in \mathcal{A}$ , extending the precondition  $\mathrm{PRE}_a$  with  $\Delta \mathrm{PRE}_a$ , denoted  $\mathrm{PRE}_a \oplus \Delta \mathrm{PRE}_a$ , is a new precondition that is satisfied in a state  $s$  if  $\mathrm{PRE}_a$  and  $\Delta \mathrm{PRE}_a$  are both satisfied in  $s$ .
Definition E.6 (Effect Extension). Given a parametric full-observable planning problem and an action  $a \in \mathcal{A}$ , extending the effect  $\mathrm{EFF}_a$  with  $(\Delta \mathrm{ADD}_a, \Delta \mathrm{DEL}_a)$ , denoted  $\mathrm{EFF}_a \oplus (\Delta \mathrm{ADD}_a, \Delta \mathrm{DEL}_a)$ , is a new effect that applied to a state  $s$  results in the new state  $s'$  that is like  $s$  but with modifications  $\mathrm{EFF}_a$ , then  $\Delta \mathrm{DEL}_a$  is not true in  $s'$ , and  $\Delta \mathrm{ADD}_a$  is true in  $s'$ , in that order, assuming that neither  $\Delta \mathrm{DEL}_a$  nor  $\Delta \mathrm{ADD}_a$  appear in  $\mathrm{EFF}_a$ .
With these operations we can define the Egocentric Planning algorithm for parametric full-observable planning problems. For instance, in the case of probabilistic planning the preconditions can be defined as in classical planning, so the precondition update is the same. The effect update of probabilistic actions is a probability distribution over the possible effects. However, extending the effect with a new effect is straightforward as the new effect must enforce that  $Pr(\mathrm{ADD}_a|s) = 1$  and  $Pr(\mathrm{DEL}_a|s) = 0$ .
For classical planning, the precondition extension is the union of the original precondition and the new precondition as both are a set of predicates. The  $\mathrm{EFF}_a$  of classical planning actions  $(\mathrm{ADD}_a,\mathrm{DEL}_a)$  is a tuple of sets of predicates, so updating the effect amounts to updating both sets of predicates.
Definition E.7 (Precondition Extension Classical). For parametric classical planning problems, extending the precondition  $\mathrm{PRE}_a\oplus \Delta \mathrm{PRE}_a$  is  $\mathrm{PRE}_a\cup \Delta \mathrm{PRE}_a$
Definition E.8 (Effect Update Classical). For parametric classical planning problems, an update to the effect  $(\mathrm{ADD}_a,\mathrm{DEL}_a)$  with  $(\Delta \mathrm{ADD}_a,\Delta \mathrm{DEL}_a)$  is  $(\mathrm{ADD}_a\cup \Delta \mathrm{ADD}_a,\mathrm{DEL}_a\cup \Delta \mathrm{DEL}_a)$ .
For ALFRED, as the planning domain in classical, ExploreAct  $(a,o)$  creates a copy of  $a$  extended with a new precondition (unknown  $o$ ) so  $a$  can only be applied if  $o$  is still unknown. The effects are extended by deleting (unknown  $o$ ) and adding (explored), so applying the exploration action satisfies the goal (explored).
F Approach: Additional Details
In this section, we provide additional details on our approach discussed in section 6. Both the environment and the symbolic planning model consider objects and parametric actions, but they are represented differently.
For instance, our approach involves a vision model that perceives in a scene objects of known classes, such as apple or fridge, including their depths and locations (see sections 4 and D). Similarly, the planning domain includes corresponding object types, such as/appleType and FridgeType (see sections 5.1 and E). When the agent perceives an apple at location 11, the algorithm creates a fresh object o1 for the apple, sets its type to (objectType o1 AppleType), and its location to (objectAtLocation o1 11). Additionally, the domain contains information about what can be done with the apple, such as (canCool FridgeType AppleType). Likewise, while the ALFRED environment contains actions like toggle, the planning domain contains actions like cool0bject, which is parameterized on arguments such as the location, the concrete object to be cooled, and the concrete fridge object. Finally, if the task provided in natural language requests to cool down an apple, the planning problem goal contains, for instance, (exists (isCooled ?o)), which is satisfied when some object has been cooled down.
The remainder of this section is structured as follows. First, we provide additional details on the Language and Vision modules, including the specific classes they detect and how they are utilized. Next, we elaborate on the semantic spatial graph and its role in generating the agent's mental model, which tracks object locations as the agent perceives and modifies the environment. We then provide specific details on the planning domain used for ALFRED, which completes the elements used in our architecture depicted in Figure 1. Finally, we conclude the section with additional details on the egocentric planning agent that won the ALFRED challenge.
F.1 Language and Vision Module
Table 7: List of Objects
alarmclock	apple	armchair	baseballbat	basketball
bathtub	bathtubbasin	bed	blinds	book
boots	bowl	box	bread	butterknife
cabinet	candle	cart	cd	cellphone
chair	cloth	coffeemachine	countertop	creditcard
cup	curtains	desk	desklamp	dish sponge
drawing	dresser	egg	floorlamp	footstool
fork	fridge	garbagecan	glassbottle	handtowel
handtowelholder	houseplant	kettle	keychain	knife
ladle	laptop	laundryhamper	laundryhamperlid	lettuce
lightswitch	microwave	mirror	mug	newspaper
ottoman	painting	pan	papertowel	papertowelroll
pen	pencil	peppershaker	pillow	plate
plunger	poster	pot	potato	remotecontrol
safe	saltshaker	scrubbush	shelf	showerdoor
showerglass	sink	sinkbasin	soapbar	soapbottle
sofa	spatula	spoon	spraybottle	statue
stoveburner	stoveknob	diningtable	coffeetable	sidetableddybear
television	tennisracket	tissuebox	toaster	toilet
toiletpaper	toiletpaperhanger	toiletpaperroll	tomato	towel
towelholder	tvstand	vase	watch	wateringcan
window	winebottle
F.1.1 Language Module
We used a BERT-based language models to convert a structured sentence into goal conditions for PDDL (DCLT19). Since the task types and object types of defined in the ALFRED metadata, we use a model to classify the type of task given a high-level language task description listed in Table 9.
Table 8: List of Receptacles
Bathtub	Bowl	Cup	Drawer
Mug	Plate	Shelf	Sink
Box	Cabinet	CoffeeMachine	CounterTop
Fridge	GarbageCan	HandTowelHolder	Microwave
PaintingHanger	Pan	Pot	StoveBurner
DiningTable	CoffeeTable	SideTable	ToiletPaperHanger
TowelHolder	Safe	BathtubBasin	ArmChair
Toilet	Sofa	Ottoman	Dresser
LaundryHamper	Desk	Bed	Cart
TVStand	Toaster
Table 9: List of Goals
pick_and_place.simple	pick_two_obj_and_place
look_at_obj_in_light	pick.cleanthen_place_in_recep
pick_heatthen_place_in_recep	pick_coolthen_place_in_recep
pick_and_place_with movable_recep
and a separate model for deciding objects and their state based on the output of the second model in Table 7 and Table 8. For example, given the task description of "put an apple on the kitchen table," the first model will tell the agent it needs to produce a "pick-up-and-place" task. The second model will then predict the objects and receptacle we need to manipulate. Then the goal condition will be extracted by Tarski and turn in to a PDDL goal like (on apple table) using the original task description and output of the both models. We use the pre-trained model provided by FILM and convert the template-based result into PDDL goal conditions suitable for our planner. Although ALFRED provides step-by-step instructions, we only use the top-level task description since they are sufficient to predicate our goal condition. Also, our agent needs to act based on the policy planner, which makes step-by-step instructions unnecessary. This adjustment could also make the agent more autonomous since step-by-step instructions are often not available in many embodied agent settings.
F.1.2 Vision Module
The vision module is used to ground relevant object information into symbols that can be processed by our planning module. At each time step, the module receives an egocentric image of the environment. The image is then processed into a depth map using a U-Net, and object maks using Mask-RCNN. (HGDG17,  $\mathrm{LCQ}^{+}18$ ). Both masks are WxH matrices where W, and N are the width and height of the input image. Each point in the depth mask represents the predicted pixel distance between the agent and the scene. The average depth of each object is calculated using the average of the sum of the point-wise product of the object and the depth mask. Only objects with an average reachable distance smaller than 1.5 meter is stored. The confidence score of which our agent can act on an object is calculated using the sum of their mask. This score gives our egocentric planner a way to prioritize which object to interact with.
F.2 Semantic Spatial Graph
The spatial graph plays the role of an agent's memory during exploration and bridges the gap between grounded objects and the planner's state representation requirements. The exploration employs an egocentric agent who only has direct access to its local space where action can be executed within a distance threshold of 1.5 meters. The key for each node is encoded as location, with visual observations being the values. The edges represent the agent's actions, comprising four specific actions: MOVEAHEAD, MOVEBACK, TURNLEFT, and TURNRIGHT, with movement actions having a step size of 0.25 meter and turning actions featuring a turn angle of  $90^{\circ}$ . This spatial graph is updated and expanded over time through policies generated by our egocentric planner, with each node storing object classes, segmentation masks, and depth information produced via the vision module. Fig. 2 shows an example of a semantic spatial graph.
Figure 2: Example of a Semantic Spatial Graph after partial Exploration. The agent is at one of the nodes.
The spatial graph undergoes continuous updates and expansions as a result of the policies generated by our egocentric planner. It involves the extraction and storage of objects within the graph, subject to a confidence score threshold of 0.8. Objects that meet or exceed this confidence score are deemed reliable and considered for storage. Each node within the graph maintains a record of various attributes associated with the objects, including object classes, segmentation masks, and depth information. The object classes provide information about the category or type of each object, allowing for efficient categorization and analysis. The segmentation masks of the extracted objects are stored as a list of pixel coordinates. These masks outline the boundaries of each object within the captured visual data. By storing the masks in this format, we can later retrieve and utilize the spatial information of each object during subsequent analyses or interactions. Moreover, the distance to each object is calculated by leveraging the predicted distance of the top 100 pixels with reference to the depth mask generated by the U-net model. The U-net model is responsible for producing a depth mask that estimates the distances of various points within the scene. By employing the predicted distance values of the top 100 pixels associated with each object, we obtain an approximate measure of its distance from the egocentric viewpoint. Actionable objects are treated as affordances at each node, allowing the agent to perform desired actions. Nodes with objects of interest are ranked based on the confidence scores outlined earlier output by the MaskRCNN.
F.3 Egocentric Algorithm for ALFRED
The egocentric planning algorithm for ALFRED uses location as the only anchor types  $\mathcal{T}_a$ , and the move action as the only exploration action  $\mathcal{X}$ . Consequently, exploration plans employ move actions to visit unexplored locations, revealing new objects visible from such locations. As the agent explores, the mental model creates new objects and adds them to the initial state, facilitating the incremental construction of a semantic spatial graph, which maps the environment and object locations. An ALFRED task,  $T_{\mathcal{E}} = \langle I_{\mathcal{E}}, G_{\mathcal{E}} \rangle$ , is characterized by a goal that corresponds to one of the seven tasks detailed in section 2, and involves specific predicates for the task-related objects. The goal is expressed as an existential formula over potential objects, as it refers to objects not visible from the initial state (Listings 2).
We also implemented some ALFRED-specific optimizations. For example, we mitigate the risk of executing irreversible actions by only attempting them after devising a plan projected to achieve the goal. This strategy implies that some actions are unavailable during the exploration phase. For each time step, our egocentric planner will produce a goal or exploration action depending on whether the agent has sufficient information to achieve its goal. The replanning loop repeats when an action fails. Priority are given to unknown location based on objects of interest for a particular task and number of total objects found in an area. They are ranked base on number of such objects. Information are gathered for each action taken which is stored in the semantic map. In order to reduce the amount of overall exploration required at the online planning stage, we allow our agent to first conduct 500 random exploration movements. The semantic graph after exploration is used as the initial state for our egocentric planning setup. To enhance efficiency, we tailor the planning domain to the specific task, disregarding actions unlikely to contribute to reasonable plans to achieve the goal. This optimization can be automatically implemented via reachability analysis, a technique commonly used in classical planning to prevent grounding actions irrelevant to achieving the goal from the initial state (Hof01).
F.4 PDDL Example
Below we provide examples assuming that the natural language description of the task is "heat potato in the microwave and put the potato on the table." We start by generating the planning setup for the egocentric planner, given the task identified as "pick-heat-then-place-in-recep." The objects involved in this manipulation are classified as "MicrowaveType", "PotatoType" and "TableType". The egocentric planner will eventually produce a sequence of actions to accomplish the desired task. In this case, the steps might include picking up the potato, heating it, and then placing it on the table.
Listings 1 and 2 show the PDDL domain and problem files for the example task. In a PDDL domain for classical planning, both predicates and actions arguments are expressed as a list of arguments, each one a variable with an optional type. For instance, (conn ?x1 - location ?x2 - location ?x3 - movement) express a predicate conn with three arguments: two locations and a kind movement: ahead, rotate-left or rotate-right. In a PDDL problem for classical planning, object constants are associated with an optional type, and the initial state is a list of predicates grounded on existing constants. For instance, l1 12 13 - location means three constants of type location. The initial state could be (and (at l1) (conn l1 12 ahead)). An action (MoveAgent ?from ?to - location ?dir - movement) has as a precondition that the agent is at the location ?from and both locations are contiguous in the direction ?dir. Executing the action sets to false that the agent is at the location ?from, and sets to true that the agent is at the location ?to. Classical plans in PDDL are sequences of actions grounded on constants. For instance, a first possible action might be (MoveAgent l1 12 ahead), leading to the state (and (at l2) (conn l1 12 ahead)). For an introduction to PDDL see (HLMM19).
Listing 1: ALFRED PDDL domain file
define (domain alfred_task)  
(:requirements :equality :typing)  
(:types  
agent - object  
location - object  
receptacle - object  
obj - object  
itemtype - object  
movement - object  
)  
(:predicates  
conn ?x1 - location ?x2 - location ?x3 - movement)  
(move ?x1 - movement)  
(atLocation ?x1 - agent ?x2 - location)  
(receptacleAtLocation ?x1 - receptacle ?x2 - location)  
(objectAtLocation ?x1 - obj ?x2 - location)  
(canContain ?x1 - itemtype ?x2 - itemtype)  
(inReceptacle ?x1 - obj ?x2 - receptacle)  
(recepInReceptacle ?x1 - receptacle ?x2 - receptacle)
(isInReceptacle ?x1 - obj) (openable ?x1 - itemtype) (canHeat ?x1 - itemtype ?x2 - itemtype) (isHeated ?x1 - obj) (canCool ?x1 - itemtype ?x2 - itemtype) (isCooled ?x1 - obj) (canClean ?x1 - itemtype ?x2 - itemtype) (isCleaned ?x1 - obj) (isMovableReceptacle ?x1 - receptacle) (receptacleType ?x1 - receptacle ?x2 - itemtype) (objectType ?x1 - obj ?x2 - itemtype) (holds ?x1 - obj) (holdsReceptacle ?x1 - receptacle) (holdsAny) (unknown ?x1 - location) (explore) (canToggle ?x1 - obj) (isTogglesd ?x1 - obj) (canSlice ?x1 - itemtype ?x2 - itemtype) (isSliced ?x1 - obj) (emptyObj ?x1 - obj) (emptyReceptacle ?x1 - receptacle)   
( :action explore_MoveAgent :parameters (?agent0 - agent ?from0 - location ?loc0 - location ?mov0 - movement) :precondition (and (atLocation ?agent0 ?from0) (move ?mov0) (conn ?from0 ?loc0 ?mov0) (unknown ?loc0) (not (explore))) :effect (and (not (atLocation ?agent0 ?from0)) (atLocation ?agent0 ?loc0) (not (unknown ?loc0)) (explore))   
( :action MoveAgent :parameters (?agent0 - agent ?from0 - location ?loc0 - location ?mov0 - movement) :precondition (and (atLocation ?agent0 ?from0) (move ?mov0) (conn ?from0 ?loc0 ?mov0)) :effect (and (not (atLocation ?agent0 ?from0)) (atLocation ?agent0 ?loc0))   
( :action pickupObject :parameters (?agent0 - agent ?loc0 - location ?obj0 - obj) :precondition (and (atLocation ?agent0 ?loc0) (objectAtLocation ?obj0 ?loc0) (not (isInReceptacle ?obj0)) (not (holdsAny))) :effect (and (not (objectAtLocation ?obj0 ?loc0)) (holds ?obj0) (holdsAny))   
( :action pickupObjectFrom :parameters (?agent0 - agent ?loc0 - location ?obj0 - obj ?recep0 - receptacle) :precondition (and (atLocation ?agent0 ?loc0) (objectAtLocation ?obj0 ?loc0) (isInReceptacle ?obj0) (inReceptacle ?obj0 ?recep0) (not (holdsAny))) :effect (and
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33   
34   
35   
36   
37   
38   
39   
40   
41   
42   
43   
44   
45   
46   
47   
48   
49   
50   
51   
52   
53   
54   
55   
56   
57   
58   
59   
60   
61   
62   
63   
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76   
77   
78   
79   
80   
81   
82   
83   
84   
85
(not (objectAtLocation ?obj0 ?loc0)) (holds ?obj0) (holdsAny) (not (isInReceptacle ?obj0)) (not (inReceptacle ?obj0 ?recep0)))   
1   
(:action putonReceptacle parameters (?agent0 - agent ?loc0 - location ?obj0 - obj ?otype0 - itemtype ?recep0 - receptacle ?rtype0 - itemtype) :precondition (and (atLocation ?agent0 ?loc0) (receptacleAtLocation ?recep0 ?loc0) (canContain ?rtype0 ?otype0) (objectType ?obj0 ?otype0) (receptacleType ?recep0 ?rtype0) (holds ?obj0) (holdsAny) (not (openable ?rtype0)) (not (inReceptacle ?obj0 ?recep0))) :effect (and (not (holdsAny)) (not (holds ?objO)) (isInReceptacle ?objO) (inReceptacle ?obj0 ?recepO) (objectAtLocation ?objO ?locO))   
1   
(:action putinReceptacle parameters (?agent0 - agent ?loc0 - location ?obj0 - obj ?otype0 - itemtype ?recep0 - receptacle ?rtype0 - itemtype) :precondition (and (atLocation ?agentO ?locO) (receptacleAtLocation ?recep0 ?locO) (canContain ?rtypeO ?otypeO) (objectType ?objO ?otypeO) (receptacleType ?recepO ?rtypeO) (holds ?objO) (holdsAny) (openable ?rtypeO) (not (inReceptacle ?objO ?recepO))) :effect (and (not (holdsAny)) (not (holds ?objO)) (isInReceptacle ?objO) (inReceptacle ?objO ?recepO) (objectAtLocation ?objO ?locO))   
1   
(:action toggleOn parameters (?agentO - agent ?locO - location ?objO - obj) :precondition (and (atLocation ?agentO ?locO) (objectAtLocation ?objO ?locO) (not (isTogglesd ?objO))) :effect (and (isTogglesd ?objO))   
1   
(:action heatObject parameters (?agentO - agent ?locO - location ?objO - obj ?otypeO - itemtype ?recepO - receptacle ?rtypeO - itemtype) :precondition (and (atLocation ?agentO ?locO) (receptacleAtLocation ?recepO ?locO) (canHeat ?rtypeO ?otypeO) (objectType ?objO ?otypeO) (receptacleType ?recepO ?rtypeO) (holds?objO)(holdsAny)) effect and
86   
87   
88   
89   
90   
91   
92   
93   
94   
95   
96   
97   
98   
99   
100   
101   
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128   
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147   
148   
149   
150
isHeated?obj0）
Next, Listings 2 shows the generated the planning problem based on the initial observation. This involves specifying the initial state of the environment and the goal that we want to achieve through the planning process.