Autonomous Learning of Object-Centric Abstractions for High-Level Planning
Authors: Steven James
Benjamin Rosman
George Konidaris
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
planning
multitask
transfer
objects

Abstract: We propose a method for autonomously learning an object-centric representation of a continuous and high-dimensional environment that is suitable for planning. Such representations can immediately be transferred between tasks that share the same types of objects, resulting in agents that require fewer samples to learn a model of a new task. We first demonstrate our approach on a 2D crafting domain consisting of numerous objects where the agent learns a compact, lifted representation that generalises across objects. We then apply it to a series of Minecraft tasks to learn object-centric representations and object types - directly from pixel data - that can be leveraged to solve new tasks quickly. The resulting learned representations enable the use of a task-level planner, resulting in an agent capable of transferring learned representations to form complex, long-term plans.

Published as a conference paper at ICLR 2022
AUTONOMOUS LEARNING OF OBJECT-CENTRES ABSTRACTIONS FOR HIGH-LEVEL PLANNING
Steven James& & Benjamin Rosman University of the Witwatersrand
George Konidaris Brown University Providence RI 02912, USA gdkêcs. brown. edu
Johannesburg, South Africa (steven. james, ben jamin. rww.sroeewwissts.ssss.
ABSTRACT
We propose method for autonomously learning an object-centric representation of continuous and high-dimensional environment that suitable for planning. Such representations can immediately be transferred between tasks that share the same types of objects, resulting in agents that require fewer samples to learn a model of a new task. We first demonstrate our approach on 2D crafting domain consisting of numerous objects where the agent learns compact, lifted representation that generalises across objects. We then apply it to a series of Minecraft tasks to learn object-centric representations and object types-directl) directly from pixel data-that can be leveraged to solve new tasks quickly. The resulting learned representations enable the use of a task-level planner, resulting in an agent capable of transferring learned representations to form complex, long-term plans.
INTRODUCTION
Model-based methods are a promising approach to improving sample efficiency in reinforcement learning (RL). However, they require the agent either learn highly detailed model-which is infeasible for sufficiently complex problems (Ho et al. 2019) or to build compact, high-level model that abstracts away unimportant details while retaining only the information required plan. This raises the question of how best to build such an abstract model. While recent advances have shown how to learn models of complex environments, they lack theoretical guarantees and require millions of sample interactions (Schrittwieser et al. 2020 Hafner et al.) [2021). Fortunately, recent work has shown how to learn an abstraction of a task that is provably suitable for planning with a given set of skills (Konidaris et al. 2018 However, these representations are highly task-specific and must be relearned for any new task, or even any small change to an existing task. This makes them fatally impractical, especially for agents that must solve multiple complex tasks.
We extend these methods by incorporating additional atructure-namdd that the world consists of objects, and that similar objects are common amongst tasks. For example, when we play video games we solve the game quickly by leveraging our existing knowledge of objects and their affordances, such as doors and ladders which occur across multiple levels (Dubey al., 2018). Similarly, robot manipulation tasks often use the same robot and a similar set of physical objects in different configurations. This can substantially improve learning efficiency, because an object-centric model can be reused wherever that same object appears (within the same task, or across different tasks) and can also be generalised across objects that behave similarly- object rypes.
We assume that the agent is able to individuate the objects in its environment, and propose framework for building portable oeject-centric abstractions given only the data collected by executing high-level skills. These abstractions specify both the abstract object attributes that support high-level planning, and an object-relative lifted transition model that can be instantiated in a new task. This reduces the number of samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously seen object types. We make the following contributions: under the assumption that the agent can individuate objects in its environment, object-centric abstractions, and for estimating object types, given only the data collected by executing high-level skills. We also show how to integrate problem-specific information to instantiate these
develop framework for building portable,

---

Published as a conference paper at ICLR 2022
representations in new task. This reduces the samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously seen objects.
We demonstrate our approach on Blocks World domain and 2D crafting domain, and then apply it to a series of Minecraft tasks where an agent autonomously learns an abstract representation of a nigh-dimensional task from raw pixel input. In particular, use the probabilistic planning domain definition language (PPDDL) (Younes & Littman (004) to represent our learned abstraction, which allows for the use of existing task-level planners. Our results show that an agent can leverage these portable abstractions to learn a representation of new Minecraft tasks using a diminishing number of samples, allowing to quickly construct plans composed of hundreds of low-level actions
2 BACKGROUND
We assume that tasks are modelled as semi-Markov decision processes M = O,S, where (i) S is the state space: (ii) (s)) is the set of temporally extended actions known as options available at state (iii) T describes the transition dynamics, specifying the probability of arriving in state s after option o is executed from 8; and (iv) R specifies the reward for reaching state after executing option in state s. An option is defined by the tuple (Io, Toi (o)). where Io is the initiation set specifying the states where the option can be executed, T, is the option policy which specifies the action execute, and 5%, the probability of the option terminating in each state (Sutton et al. 1999).
We adopt the object-centric formulation from Ugur & Piater (2015): in a task with n objects, the state is represented by the set (fa, f1 fz. .........5... ]. where fa is vector of the agent's 's features and f, is vector of features particular to object i. Note that the feature vector describing each object can itself be arbitrarily complex, such as an image voxel grid-in one of our domains we use pixels.
Our state space representation assumes that individual objects have already been factored into their constituent low-level attributes. Practically, this means that the agent is aware that the world consists of objects, but is unaware of what the objects are, or whether multiple instantiations of the same object are present. It also easy to see that different tasks will have differing numbers of objects with potentially arbitrary ordering; any learned abstract representation should be agnostic to this.
2.1 STATE ABSTRACTIONS FOR PLANNING
We intend to learn abstract representation suitable for planning. Prior work has shown that sound and complete abstract representation must necessarily be able to estimate the set of initiating and terminating states for each option (Konidaris et al.) 2018). In classical planning, this corresponds to the precondition and effect of each high-level action operator (McDermott et al. 1998).
The precondition is defined as Pre(o) Pr(s 1.), which is probabilistic classifier that expresses the probability that option o can be executed at state s. Similarly, the effect or image represents the distribution of states an agent may find itself in after executing an option from states drawn from some starting distribution (Konidaris et al., 2018). Since the precondition is probabilistic classifier and the effect is density estimator, they can be learned directly from option execution data.
We can use preconditions and effects to evaluate the probability of sequence of options plan executing successfully. Given an initial state distribution, the precondition used to evaluate the probability that the first option can execute, and the effects are used to determine the resulting state distribution. We can apply the same logic to the subsequent options to compute the probability of the entire plan executing successfully. It follows that these representations are sufficient for evaluating the probability of successfully executing any plan (Konidaris et al. 2018).
Partitioned Options For large or continuous state spaces, estimating Pr(s' | s, s, is difficult because the worst case requires learning distribution conditioned on every state. However, if we assume that terminating states are independent of starting states, we can make the simplification Pr(s') 0) Pr(s' o). These subgoal options (Precup) 2000) are not overly restrictive, since they refer to options that drive an agent to some set of states with high reliability. Nonetheless, many options are not subgoal. It often possible, however, to partition an option's 's initiation set
Mooe results and videos can be found at: https://siitei googl e www.///vviieve ne-p
2

---

Published as a conference paper at ICLR 2022
into finite number of subsets, so that it is approximately subgoal when executed from any of the individual subsets. That is, we partition an option o's start states into finite regions such that Pr(s' |, 0) x Pr(s' o, o) C (Konidaris et al. 2018).
Factors We adopt the frame assumption-an assumption implicit in classical Haaning-which states that aspects of the world not explicitly affected by an agent's action remain the same (Pasula al., 2004). Prior work leverages this to learn a factored or STRIPS-like (Fikes & Nilsson 1971) representation by computing the option's 's mask: the state variables explicitly changed by the option (Konidaris et al. 2018). our formulation, the state space already factorised into its constituent objects, computing the mask amounts to determining which objects are affected by given option.
LEARNING CINECT-ENTTIC REPRESENTATIONS
Although prior work (Konidaris et al. 2018) allows an agent to autonomously learn an abstract representation supporting fast task-level planning. that representation lacks generalisability- the symbols are distributions over states in the current task, they cannot be reused in new ones. This approach can be fatally expensive in complex domains, where learning an abstract model may be as hard as solving task from scratch, and is therefore pointless if we only want to solve a single task. However, an agent able to reuse aspects its learned representation can amortise the cost of learning over many interactions, accelerating learning in later tasks. The key question is what forms of representation support transfer this way.
We now introduce an object-centric generalisation of a learned symbolic representation that admits transfer in tasks when the state space representation consists features centred on objects in the environment. This is common in robotics, where each object is often isolated from the environment and represented as a point cloud or subsequently a voxelised occupancy grid. Our approach builds on significant amount of machinery, involving clustering, feature selection, classification and density estimation. We summarise our proposed approach in Figure the remainder of this section, but provide pseudocode and specific practical details in the appendix.
[ and provide a high-level description in
Step Par- tition options into subgoal options
Step 2: Generate propositional forward model
Step 4: Lift abstractions based on object type
Step 5: Instantiate ab- stractions for current task
Step 3: Merge objects into types
Figure 1: Learning lifted representations from data. Red nodes represent problem- -specffic representa- tions, while green nodes abstractions that can be transferred between tasks.
3.1 GENERATING PROPOSITIONAL MODEL (STEPS 1-2) (AS IN KONIDARIS ET AL... 2018
The agent begins by collecting transition data using an exploration policy. The first step to partition the options into approximately subgoal options. For each option o and empirical sets of initial and terminating states I, and B., the agent partitions I, into number of disjoint subsets. such that for each subset K C I.. we have Pr(s' s; o) Pr(s' ( o)Vs; Sj E K, E B... In other words, the effect distribution of the option identical, independent of the state in K from which it was executed. In practice, this can be approximated by first clustering state transition samples based on terminating states, and then assigning each cluster to a partition. Finally, pairs of partitions whose initiating states overlap are merged to handle probabilistic effects (Konidaris et al. 2018).
The agent next learns a precondition classifier for each approximately partitioned option. For each partition, the initiating states used as positive examples, and all other states treated as negative ones. A feature selection procedure next determines which objects are relevant to the precondition, and a classifier is fit using only those objects. A density estimator is used to estimate the effect distribution for each partitioned option; the agent learns distributions over only the objects affected by the option. Together these effect distributions form our propositional PPDDL vocabulary V.
To construct a PPDDL representation for each partitioned option, we must specify both the precondi- tion and effects using the state distributions (propositions) in V. The effects are directly specified
3

---

Published as a conference paper at ICLR 2022
using these distributions, and pose no problem. However, the estimated precondition is a clas- sifier rather than a state distribution. The agent must therefore iterate through all possible effect distributions to compute whether the skill can be executed there. To do so, we denote P as some set of propositions in V, and (s:; P) as the probability that a low-level state 8 is drawn from the conjunction of propositions in P. Then, for an option with learned classifier fo, we can represent the precondition with every P с (P()) such that Is (=)) P)ds > O, where (?()) denotes the powerset of V. In other words, the agent considers every combination of effect state distributions and draws samples from their conjunction. these samples are classified as positive by Lo. then the conjunction of P is used to represent the precondition. The preconditions and effects are now specified using distributions over state variables, where each distribution is a proposition this is our PPDDL representation, which is sound and complete for planning.
3.2
GENERATING LIFTED, TYPED MODEL (STEPS 3-4)
At this point the agent has learned an abstract, but task-specific, representation. Unfortunately there no opportunity for transfer (both within the task and between different tasks), because each object is treated as unique. To overcome this, we now propose a method for estimating object types using the PPDDL preconditions and effects learned in the previous section.
Definition Assume that option o has been partitioned into n subgoal options Object i's profile under option o is denoted by the set
Profile(i, o) = (Prre ",,"")
(Pre, ((n) 1).0)))
where Pre; (k) is the distribution over object i's states present in the precondition for partition k, and go(k) is object i's effect distribution. Note that these preconditions and effects can be null.
Definition 2. Two objects i and j are option-equivalent if, for given option o, Profile o) Profile(j: o). Furthermore, two objects are equivalent if they are option-equivalent for every o in O.
The above definition implies that objects are equivalent if one object can be substituted for another while preserving every operator's abstract preconditions and effects. Such objects can be grouped into the same object type, since they are functionally indistinguishable for the purposes of planning. In practice, however, skill will usually modify only the object being acted upon, and because we have subgoal options, we can take similar approach to Ugur & Piater (2015) and group objects by effects only:
weaker condition to construct object types. Since object-centric
Definition 3. Assume that option has been partitioned into n subgoal options .())....((n)). Object i's effect profile under option o is denoted by the set
EffectProfile(i,.) o) ( g0(')) },
where (,()) is object i's effect distribution. Two objects i and are effect-equivalent if EffectProoileei.. o) = EffectProfilejj.0 o) for every o in O.
The notion of effect equivalence was first proposed by Sahin et al. (007). Such an approach assumes that an object's type depends on both the intrinsic properties of the object, as well as the agent's endowed behaviours (Chemero) 2003). However, this definition does not account for preconditions, nor does it consider interactions involving multiple objects. The latter issue is also present in frameworks such as object-oriented MDPs< et al. (208),, where the dynamics are described by pairwise object interactions (one of those objects being the agent). Nonetheless, the approach will prove sufficient for our purposes, and we leave a more complete definition to future work.
By computing effect profiles using the propositional representation, the agent can determine whether objects i and j are similar (using an appropriate measure of distribution similarity) and, if so, merge them into the same object type. Propositions representing distributions over individual objects can now be replaced with predicates that are parameterised by types. For example, consider a domain with three objects- titects-twooot two identical doors and a block and an agent with a single option to open a door. Since the option can affect both of the doors, it would first be partitioned into two subgoal options (one for each door). Given this, the effect profile for the first and second doors would be {open1, 2 and (00, 2 respectively. The effect profile for the block, which cannot be acted upon, would
4

---

Published as a conference paper at ICLR 2022
simply be {2, 2). By noting that the sets representing the doors' effects equal, the agent can merge the two door objects into a single type and replace the openl and open. propositions with a single open predicate parameterised by objects of type door.
3.3 SOORLEM-SPECIREE INSTANTIATION (STEP 5)
If the task dynamics are completely described by the state of each object, is the case in object- oriented MDPs (Diuk et al. 22088) then our typed representation is sufficient for planning. However, in many domains the object-centric state space is not Markov. For example, in a task where only a particular key opens a particular door, the state of the objects alone is insufficient to describe dynamics the identities of the key and door are necessary too. common strategy in this case is to augment an ego- or object-centric state space with problem-specific, allocentric information to preserve the Markov property (Konidaris et al.) 2012 James et al. 2020). We denote as the space of problem-specific state variables. S remains the original object-centric state space. The above complication does not negate the benefit of learning transferable abstract representations, existing operators learned in S can be augmented with propositions over X' on per-task basis. In general, local information relative to individual objects will transfer between tasks, but problem-specific information, such
object's global location, must be relearned each time.
For given partioned option, the agent repeats the partitioning procedure, but this time using only problem-specific state data. This forms n partitioned options that are subgoal in both S and X. Denote K; and X; for E (1,..., ni as the sets of start and end states for each of these newly partitioned options. The agent can now ground the operator by appending each K2 and X to the precondition and effect, treating each K; and X as groblem-speciffe propositions. Finally, these sppeeic-ppecffic propositions must be linked with the grounded objects being acted upon. The agent therefore adds precondition predicate conditioned on the identity of the grounded objects, where objects' identities are simply their respective index in the state vector.
4 EXPERIMENTS
We first demonstrate our framework on the classic Blocks World domain (Section 4.1
While the
high-level operators and predicates describing the domain are usually given, we show how such representation can be learned autonomously from scratch. In Section 4.2 we apply our method to 2D crafting environment (Andreas al. (017) consisting of over 20 objects, where an agent learns representation that generalises across objects and can be used to solve hierarchical tasks. We then demonstrate that our method scales to significantly harder problems by applying to a high- dimensional Minecraft task (Section 4.3). Finally, we investigate the transferability of the learned abstractions by transferring them to additional procedurally generated Minecraft tasks (Section 4.4). Owing to space constraints, defer the exact implementation and domain details to the appendix.
4.1 LEARNING A REPRESENTATION OF BLOCKS WORLD
Blocks World consists of several blocks which can be stacked on top of one another by an agent (hand). The agent possesses options that allow it to pick up a block (P (Pick). put a block back on the table (Put), and stack one block on another (Stack). Blocks cannot be picked up if they are covered or if the hand is occupied, and can only be put down or stacked if already gripped. We consider the task consisting of three blocks A, B and C, where each block is described by whether there nothing, another block, or table directly above or below it. This allows us to determine whether given block is on a table, on another block, or in the hand, and similarly whether another block has been stacked upon it. The hand is characterised by a Boolean indicating whether it is holding a block. Thus a state is described by [fH, fA. fp. fc). corresponding to the hand and blocks' features respectively.
Generating a Propositional Model (Steps 1-2) Using the approach outlined in Section 3.1 the agent partitions the options using transition data collected from the environment. This results in a total of 30 partitioned options. It then fits a classifier to each partition's 's initiation states, and a density estimator to its terminating states. Finally, the agent generates propositional PDDL using these learned preconditions and effects. The full PDDL (leared entirely from data), as well as an illustration of learned propositional operator, is provided in the appendix.
5

---

Published as a conference paper at ICLR 2022
Generating Lifted Typed Model (Steps 3-4) Using the effects from the propositional represen- tation, the agent determines that objects A, B and all possess the same effect profiles for all options and can be grouped into a single type, while the hand belongs to its own type. The agent can now lift its representation by replacing the learned propositions with predicates parameterised by the above types. For example, after generating the model, there three propositions: AOnTabl e, BOnTable e, and COnTable Since these are distributions over objects determined to be the same type, the agent can replace them all with single predicate TaTable (X), which accepts block objects. As result, the agent can reduce the number of operators from 30 to 6, resulting in more compact representation with a smaller branching factor. Figure 2 illustrates a lifted operator for picking any block X off any block Y, while the full parameterised PDDL is listed in the appendix.
(saction #ick-partition-10 :porametera 17N type0 typel
7y type")
(a) symbol_4 4
(b) symbol.5
(c) symbol.2
:precondition (and (symbol_4 ?w)
(symbol_5 ?x] taymon ?y1
reffect (and (symbol_ 0 Pyl (eymbol 2x) (symbol_1 2w) (not (aymbol_2 by)1
(not (symbol_4 220) (not (symbol_s 2x1))
(d) symbol.0
symbol 3
(I) symbol
The learned lifted operator for a Pick action describing picking a block off another, In order pick up block Y, it must be on block which itself is on the table, and the hand must be empty. As result, the hand is not empty, Y is now in the hand, and x is on the table and clear. type0 refers to the "hand" type, while type1 refers to the "block" type.
Figure 2:
4.2 LEARNING A REPRESENTATION OF CRAFTING TASK
To demonstrate the applicability of our approach to tasks with a large number of objects, we extend the domain proposed by Andreas al. (017)) where an agent required to collect objects and craft them to create new ones. The domain consists of a 12 x 12 grid containing multiple objects, such as wood, grass, iron, rock, water and gold, as well as three workshops. Other items such as bridges can be crafted using collected objects, but only if the agent is standing near the appropriate workshop. The state of each object characterised by whether by the agent (pi cked) or does not yet exist (non-ex iexissiseeee stent); the state of the world is given by the state of each object (of which there are 23), as well as the agent's egocentric view. The agent is given four skills: WalkTo, Pickup, P Place and Craft-an object can only be picked up when the agent is adjacent to it, and any particular item can only be crafted the agent has picked up the correct "ingredients". Furthermore, the gold object is separated by water and can only be accessed once the agent has placed a bridge, which must first be crafted. To solve the task, the agent is required construct a gold arrow, which involves collecting wood and iron to create bridge, then collecting the gold, crafting a stick out of wood, and then finally crafting the arrow using the stick and gold. The optimal plan consists of 15 option executions, portion of which is illustrated by Figure 3a
on the grid (present), ), has been picked up
Generating a Propositional Model (Steps 1-2) The agent executes options uniformly at random to collect 20 episodes' worth of transition data. The partitioning procedure results in 20 Wal 1 kTo options, 17 Pickup options, Place option and 3 CraftOptionn ions. For each partitioned option, the agent estimates the preconditions and effects, which are then used to construct propositional PDDL. consisting of 192 action operators.
Generating Lifted, Typed Model (Steps 3-4) Using the effects from the propositional represen- tation, the agent next groups objects into types based on their effect profiles. Since our approach to estimating types depends on the agent's 's actions, we do not recover the ground truth types such as grass and iron. Instead we discover six types: type0 represents the agent, while typel represents objects that can be picked up and used in crafting other objects. type2 represents objects which can be picked up but are not used in crafting. This occurs only because the agent does not observe particular objects being used in this manner during the 20 episodes. The three workshops that cannot be manipulated form type3, while t type4 are those objects that can be crafted. Finally, type5 represents the bridge, which is the only object that can be placed.

---

Published as a conference paper at ICLR 2022
Problem-Specific Instantiation (Step 5) Since the learned representation is object-centric, it can be reused for different configurations of the domain, with varying numbers of objects. Unlike the Blocks World domain, however, the identities of the objects here important. For example, the bridge can only be crafted at workshop2, and so the workshops cannot be treated interchangeably. To overcome this, the agent is required to ground the learned operators in the current task using the object identities (which can be determined using the option masks). An example of a learned, grounded operator given by Figure 3 For any new task, the agent need only determine the identity and type of each object, which can then be combined with the learned operator to create PDDL representation suitable for planning. Finally, we apply the miniGPT planner (Bonet & Geffner 2005) which uses the learned representation to discover an optimal plan. More results, types and associated objects, and visualised operators are provided in the appendíx.
WalkTa iron
symtboLO
sysabol
Pickup iron
WalkTo wood
Pickup wood
symbol44
WalkTo workshop1
Craft bridge
(ract walk pactition
Place bridge
sparansteza (2a type0 Eypel 20 typas)
sprecardit ion (and Daynbal__ 7a) [synbol_4 760
WalkTo gold
(ayolol1_4 he) 1= rid 7ay 10)
Pickup gold
sacfact (and tayabol_33
(synbol_D
(a)
(b)
Figure 3: (ap Simplified illustration of the domain. Here, the agent executes series of options that involves crafting a bridge to allow it to collect the gold. The full domain consists of 23 objects (excluding the agent). (b) An operator for walking to the gold. To execute, the agent must be standing in front of empty space (symbol. 0), the gold must be present (s (symbol 4) and the bridge placed (s (symbol 1 1.41). As a result, the agent finds itself in front of the gold (symbol.333 The red symbol is the only predicate that must be relearned in any new configuration.
4.3 LEARNING A REPRESENTATION OF MINECRAFT TASK
In the above examples, objects are represented using pre-specified features; however, our approach capable of scaling beyond this simple case and learning these features from pixels. We now demonstrate this in a complex Minecraft task (Johnson et al. 2016)) consisting of five rooms with various items positioned throughout. Rooms are connected with either regular doors which can be opened by direct interaction, or puzzle doors requiring the agent to pull lever to open. The world is described by the state of each of the objects (given directly by each object's appearance as a 600 x 800 RGB image), the agent's 's view, and current inventory. To simplify learning, downscale images and applying PCA (Pearson 1901) to a greyscaled version, preserving the top 40 principal components.
The agent is given high-level skills, such ToggleDoor and WalkToItem: Execution is stochastic- opening doors occasionally fails, and the navigation skills are noisy in their execu- tion. To solve the task, agent must first collect the pickaxe, use it to break the gold and redstone blocks, and collect the resulting items. must then navigate to the crafting table, where it uses the collected items first craft gold ingots and subsequently clock. Finally, it must navigate the chest and open it to complete the task. This requires a long horizon,, hierarchical plan-the tte shortest plan that solves the task consists of 28 options constituting hundreds of low-level continuous actions.
Generating Propositional Model (Steps 1-2) The agent first learns model using the method outlined in Section [.1 The agent partitions options using DBSCAN (Ester et al., 1996) to cluster option data based on terminating states. For each partitioned option, it then fits an SVM (Cortes) & Vapnik 1995) with Platt scaling (Platt 1999) to estimate the preconditions, and a kernel density estimator Rosenblatt 1956) for effects, which are then used to construct the propositional PPDDL.
Generating a Lifted, Typed Model (Steps 3-4) Using the effects from the propositional represen- tation, the agent groups objects into types based on their effect profiles. This is made easier
7

---

Published as a conference paper at ICLR 2022
because certain objects are not affected by all options. For example, the chest cannot be toggled, while a door can, and thus is immediately clear that they are not of the same type. Having deter- mined the types, the agent replaces all similar propositions (where similarity is measured using the -il-diveggence) with a single predicate parameterised by an object of that type.
Problem-Specific Instantiation (Step 5) The agent now has representation whose operators be transferred between tasks. However, complication arises because the object-centric state space is Markov. For example, state where all the doors are closed and the agent is in front of the first door is indistinguishable from state where the agent is in front of the second door. As described in Section 3.3 the agent must ground the representations the current task by incorporating additional cobllem-speciti state variables to preserve the Markov property. These state variables are fixed across the family of MDPs; in this case, they are the agent's myz-location.
For each partitioned option, the agent again uses DBSCAN to cluster end states X' to form partitioned subgoal options in both S and X'. Each of these clusters A' is problem-specific proposition, which can be added to the learned operators to ground the problem. Figure illustrates iillstaatlss learned operator for opening particular door, where the problem-specifee symbol has been tied to the door being opened in this manner. Without modifying the operator's parameter, it would be possible to open any door at that location. The final plan discovered by the agent is illustrated by Figure 14 in Appendix K
(saction Toggle Dooz partition la :parameters 12w type0 :pzecond_tion fand (notCalled) (symbol_37
typel)
(symbol 2x1
[= (id ?x)) 1) (psymbol_241) reffect [and (aymbo 7x)
(symbol_66 65 ?w). (not (ayabol_9 78)
(a) symbol 37
(b) symbol 9
(c) psynbol 24
(not (symbol_37 2w)1)
(I) A learned typed PDDL operator for one partition of the Goggle-Dooo option. The predicates underlined learned for each new task, while the rest the operator can be safely transferred.
red must be re-
(d) symbol. 64
(e) symbol. 1ombol..63. 655
Figure To open a particular door, the agent must be standing in front of a closed door (symbol_37) particular location (psymbol. 1.24) 24.. and the door must be closed (symbol.9) The effect of the skill is that the agent finds itself in front of an open door (symbol.66 (symbol_65) typeO and typel refer to the "agent" and "door" classes, while id is fluent specifying the identity of the grounded door object, and is linked to the problem-specific symbol underlined in red.
and the door is open
4.4 INTER-TASK TRANSFER IN MINECRAFT
We next investigate transferring operators between five procedurally generated tasks, where each task differs in the location of the objects and doors; the agent cannot thus simply plan found in one task to solve another. For given task, the agent transfers all operators learned from previous tasks, and continues to collect samples using uniform random exploration until produces a model which predicts that the optimal plan can be executed. Figure shows the number of operators transferred between tasks, and the number of samples required learn model of a new task.
The minimum number of samples required to learn a model for a new task is bounded by the exploration strategy, since we must discover all problem-specinn symbols to complete the model. Figure 5b shows that the number of samples required to learn model decreases over time towards this lower bound. Inter-task transfer could further improved by leveraging the agent's existing knowledge perform non-umt in-unfoorm exploration, but we leave this to future work.
5 RELATED WORK
boddel-based RL methods have previously been applied to pixel-based domains. For example, Kaiser al. (2020) and Haffnr et al. (202)) learn forward model in latent space, while Eysenbach et al.
8

---

Published as a conference paper at ICLR 2022
60
Number df Option Executonns
950
ntation only
800
40
30
No Transter
650
Number
20
500
10
350
200
3 Number Tasks Seen
5
0
Number
Tasks Seen
(a) Orange bars represent the number of opera- tors that must be learned by the framework of Konidaris al. (2018) to produce sufficiently ac- curate model solve the task. Blue bars represent the number of operators transferred between tasks. As the number of tasks increases, the number of new operators that must be learned decreases.
(b) Number of samples required to learn suffi- ciently accurate models as function of the number of tasks encountered. The red line represents the number samples required learn all the oper- ators and the instantiation, while the green line accounts for the instantiation phase only.
Figure 5: Results of learning and transferring high-level abstractions between tasks. We report the mean and standard deviation averaged over 80 runs with random task orderings.
(019)) computes planning graph over the buffer of observed states, and Srinivas et al. (2018) learn goal-directed latent space in which planning occur. However, all of these approaches require hundreds of thousands of samples, which cannot feasibly be collected using the Malmo platform. By contrast, we are able to construct an abstract model using very little data.
Our method is an extension of the framework proposed by Konidaris et al. 2018) which does not allow for the learned representations be transferred between tasks. James agent-centric PPDDL representation which can be transferred between tasks. but the operators have no notion of objects or types, limiting generalisability. As such, the method scales poorly with an increase in the number of objects, since their representations must be learned from scratch. In contrast, by assuming the existence of objects, we are able to learn and transfer representations across tasks that share similar types of objects.
(220)) learn an
There has been work autonomously learning parameterised, transferable representations of skills from raw data. Ugur & Piater (2015) learn object-centric PDDL representations for robotic object manipulation tasks. Similarly to our work, they estimate object types by clustering objects based on how actions affect their states, but the object features specified prior to learning, and discrete relations between object properties are given. Furthermore, certain predicates are manually inserted generate sound representation. Asai (2019) learns object-centric abstractions directly from pixels, but the representations are encoded implicitly and cannot be transformed into a language that can be used by existing planners. This limitation is removed by Asai & Muise (022)), who propose an approach to extract PDDL representations from image-based transitions. However, their framework does not provide soundness guarantees, nor does consider stochastic dynamics. By contrast, we are able to learn object-centric representations, along with the object types and the abstract high-level dynamics model, directly from raw data in form that can be used by off-the-shelf planners.
CONCLUSION
We have introduced method for learning high-level, object-centric representations that can be used by task-level planners. In particular, we have demonstrated how to learn the type system, predicates, and high-level operators all from pixel data. Our representation generalises across objects and can be transferred to new tasks. Although we have injected structure by assuming the existence of objects, this reflects the nature of many environments: fields such as computer vision assume that the world consists of objects, while there is evidence to suggest that infants do the same (Spelke) 1990). This allows us to convert complex, high-dimensional environments to abstract representations that serve as input to lesk-level planners. Our approach provides an avenue for solving pparse-reware long-term planning problems- such as the MineRL competition (Guss et al. 2019) that are beyond the reach of current approaches.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This research was supported by the National Research Foundation (NRF) of South Africa under grant number 117808, and by NSF CAREER Award 1844960, DARPA under agreement numbers W91 INF INF1820266 and (YFA) D15AP00104, AFOSR YIP agreement number FA9550-17- 17-1-0124., and the NSF under agreement number 1955361. The U.S. Govermment authorised to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The content solely the responsibility the authors and does not necessarily represent the official views of the NRF, NSF, DARPA, the AFOSR.
The authors acknowledge the Centre for High Performance Computing (CHPC), South Africa, for providing computational resources this research project. Computations were also performed using High Performance Computing Infrastructure provided by the Mathematical Sciences Support unit at the University of the Witwatersrand.
REFERENCES
B. Ames, A. Thackston, and G.D. Konidaris. Learning symbolic representations for planning with parameterized skills. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018
G. Andersen and G.D. Konidaris. Active exploration for learning symbolic representations. In Advances in Neural Information Processing Systems, pp. 5016-5026, 2017.
Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pp. 166-175. PMLR, 2017.
M. Asai and C. Muise. Learning neural-symbolic descriptive planning models via cube-space priors: The voyage home (to strips). In Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020.
Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 583-591, 2019.
B. Bonet and H. Geffner. mGPT: A probabilistic planner based heuristic search. Journal of Artificial Intelligence Research, 24:933-944, 2005.
A. Chemero. An outline of theory of affordances. Ecological Psychology. 15(2):181-155. 2003..
C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273-297 1995.
C. Diuk, A. Cohen, and M.I. Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference on Machine Learning, pp. 240-247, 2008.
R. Dubey, P. Agrawal, D. Pathak, T.L.. Griffiths, and A.A. Efros. Investigating human priors for playing video games. In International Conference on Machine Learning, 2018.
M. Ester, H. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. 2nd International Conference on Knowledge Discovery and Data Mining, volume 96, pp. 226-231, 1996.
B. Eysenbach, R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. arXiv preprint arXiv: 1906 2019.
R.E. Fikes and N.J. Nilsson. STRIPS: A approach the application of theorem proving problem solving, Artificial Intelligence, 2(3-4):189-208, 1971.
W. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. Liebana, R. Salakhutdinov, N. Topin, et al. The MineRL competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv: 10079, 2019.
10

---

Published as a conference paper at ICLR 2022
D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering Atari with discrete world models. In International Conference on Learning Representations. 2021.
Mark K Ho, David Abel, Thomas Grifftth, and Michael L. Littman. The value of abstraction.
Current Opinion in Behavioral Sciences, 29:111-116, 2019.
S. James, B. Rosman, and G.D. Konidaris. Learning to plan with portable symbols. In International Conference on Machine Learning, 2020.
M. Johnson, K. Hofmann, T. Hutton, and D. Bignell. The Malmo platform for artificial intelli- gence experimentation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pp. 4246-4247, 2016.
L. Kaiser, M. Babacizadch, P. Milos, B. Osinski, R. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, Levine, et al. Model-based reinforcement learning for Atari, In International Conference Learning Representations, 2020.
G.D. Konidaris, I. Scheidwasser, and A.G. Barto. Transfer in reinforcement learning via shared features. Journal of Machine Learning Research, 13(May):1333-133) 2012.
G.D. Konidaris, L.P. Kaelbling, and T. Lozano-Pérez. From skills symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61 (January):215-28) 2018.
D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL-the planning domain definition language, 1998.
H. Pasula, L.S. Zettlemoyer, and L.P. Kaelbling. Learning probabilistic relational planning rules. In Proceedings of the Fourteenth International Conference Automated Planning and Scheduling. pp. 73-81, 2004.
K. Pearson. On lines and planes of closest fit to systems points space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):55--572, 1901.
J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10())61174, 1999.
D. Precup. Temporal abstraction in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 2000.
N. Rosenblatt. Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics, pp. 832-837, 1956.
E Şahin, M. Cakmak, M. Doğar, E. Uğur, and G. Üçoluk. To afford or not to afford: A new formalization of affordances toward affordance- robot control. Adaptive Behavior, 15(4): 447-472 2007.
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, al. Mastering Atari, Go, chess and shogi by planning with learned model. Nature, 588(7839) 2020.
E. Spelke. Principles of object perception. Cognitive Science, 14(1):29-55) -62 1990.
A. Srinivas, A. Jabri, Abbeel, Levine S., and Finn C. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference Machine Learning, pp. 4732-4741, 2018.
R.S. Sutton, Precup, and Singh. Between MDPs and semi-MDPs: framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 11((12))181-211, 1999.
E. Ugur and J. Piater. Bottom-up learning of object categories, action effects and logical rules: from continuous manipulative exploration symbolic planning. In Proceedings of the 2015 IEEE International Conference on Robotics and Automation, pp. 2627-2633, 2015.
H.L.S. Younes and M.L. Littman. PPDDL 1.0: An extension to PDDL for expressing planning domains with probabilistic effects. Technical report, 2004.
11

---

Published as a conference paper at ICLR 2022
A ENUMERATING SUBGOAL OPTIONS FOR THE BLOCKS WORLD DOMAIN
Given the description of the Blocks World domain in the main text, we must partition the given options (P i ick, Put and Stack) so that they adhere the subgoal condition. When there are three blocks in the environment, we see that there are 30 partitioned options, which are described in the table below:
Option
partitions
Description of start states
Description of end states
PickoffTable (X)
3
X is on the table, x is clear, and the hand empty.
X is grasped in the hand.
PickOffSingle5locck (x, Y)
X on block Y which is on the table, x is clear, and the hand empty.
X is grasped in the hand and Y is clear and on the table.
PickOffDoubleslocc (x, Y)
X on block Y which is another block, is clear, and the hand is empty.
X is grasped in the hand and Y is clear and on another block.
StackonSingleBlockk (x, Y)
X is in the hand, and Y is clear and on the table.
X is on block Y which on the table, and the hand is empty. is on block Y which is on another block, and the hand is empty.
StackOnbouuleellocck (x, Y)
is in the hand, and Y is clear and on an- other block.
Put (X)
X grasped in the hand.
X is on the table and the hand is empty.
Table 1: Descriptions of the different option partitions. The description start and end states includes only the relevant information.
B PROPOSITIONAL PDDL DESCRIPTION FOR THE BLOCKS WORLD TASK
Below is the automatically generated propositional PDDL description of the Blocks World do- main with blocks. In practice, the agent generates this description with arbitrary names for the propositions, but for readability purposes we have manually renamed them to match their semantics.
(donain BlocksWor1d)
(:cequicemeats :strips)
dspredicates
(ootfilled)
(AInHand)
(HladFulll)
(C0nBlock)
(BIalland)
(COnTablle
(AOnTable)
(o0nRlock)
(A0nBlock)
(30nTable)
(CIalland)
[HeddBmpty]
(s0nTable__d) (
[B0nBlock_
(coniable_Ccvvere
[Aovtoble_ACovrred)
12

---

Published as a conference paper at ICLR 2022
(raction
sparsmeters()
:precondition land HandEmppy) (AOnTable) (nottalled)) reffect (and (AInlland) (BandFulll) (not AOnTable) (zot [lanEEoot)) (not AOnTable) (net HendEmpty!)
(raction sparsmeters
:precondition land HHlandEppcy) (AOnBlock) SeaBlockkCCovereec (not(alled)) reffect (end (COnBlock) (AInlland) (landFull) (net a0nBlock) (nct HandEmpty) (nct CUmblock_ clmslock_oCovered) (not Conklock, (akklook_Ccoeeee)
(Covered) (not Adnslock) (not Handkmpsy)
(saction Pick_2 :pacemeters(1 sprecendition land (Handimpty) Connaabl__Covereee 180n5leck) (notfailed)) reffect (and (BinHand) (contable) (HandFull) (not sCnslock) (nct. Hondimpty) (not ConTable (_tabble_ccoeeed) BonSlook) (not HandRmpty) (not SonTable_CCovezedl)
(raction Pick_3 sparemeters sprecondition land [HandRmpty) (AOnTabla, ACovered:
(80nslook) (cooerel (notfai led)
reffect
(BInfland) (NOnTable) (HandFull) B0aBlock) HandEmpty) (not 3CnIable_ACoverze (not DOnBlock) (not HanddEpp-y) adhitbll__Coveeedd))
(saction Bick_4
sparanetera gprecondition IllandEmply) (AOnBlock) BeuBlocck_Bovereee (nol(ailed) reffect (end (BCnBlock) (Ainlland) (HandFull) (not #CaBlock) (nct HandEmpty) (nct sonBlock (asllock_Bcoered) (not Aonslosk) (not Handimpty) (nct Boobecck_BCovered))
(saction Pick_5 :pacemeters sprecondition land (Handkmpzy) IkonsiockAAoveeree 180nslcsk)) (natfailed))
seffect
(RInHand) (AonBlack) (#ondFul1) (not scmBlock) (not HandEmpty) (nct Acoveec)_ACovered) (not 20nBlock) (not HandEmpsy) (not Adnslock_AAovered)))
(saction
sparcmeters(!
:precondition land [HandRmpty) (AOnBlock) (ROrTable, (Rortable_RCovared) (not led))
reffect (end (HUnrablel (Ainsand) (sendrell) (not aunslock) (nct Handsmpty) (not sonTable_ Sentable__Cavered) sontable_BCovered)
SCovered) (not A0nSloak) (not HandEmpty)
(saction pick_?
sparametera(i sprecendition land [Handkmpty) (8OnTable) (natfailed))
seffect (and (RTnRand) (HandFull) inot RonTable) (not HandFmpty) (not ROnTTaa e) (not HandEmpty!)
(saction Pick_8 sparametera(1 pprecondition land IllandEmpty) (COnTable) (not.falled))
reffect (and (CInflane) (HandFull) inot conTable) (not HancEmpty) (not conTagle) HendEmpty!)
(sactionn Pick_9 spacemeters(1 sprecondition land [HladEmp=y) Acoveeei ICOnBleck) (notfailed)) reffect (and (CInfland) (AOnTable) (HaddFulll (not ConBlock) (not HandEmpty) AdnTable_ACoveze (net xcoveebe
cOnBlock) (not HandDmppy)
(saction 11 :paremeters
sprecondition land IllandEapty) (AOnBlock) (COnTable_CCoveel (not(elled) reffect (CCntable) (Alnllan)) (HandFull) ACnBlock) Handimpty)
13

---

Published as a conference paper at ICLR 2022
Contable__Coverre (hot contabble_CCaveeedd)
AOnslock) (not Handimp=y)
(raction #ick_11 sparaneters sprecondition land [HandRmpty) (AOnBlock ACoveredi Mooveeelk_Cooerred 1C0nBlook) (CinHand) (ADnslack) (8andFull) (nct ConBlock) Handimpty) (not ACnB BaBllock_aa ook_ACovered) (not cOnRlook) (not HandRmpty) (not AOoBLock_ACovered)
(not led))
(saction Fick_12 tparametera :precendition [HandRmptyl (conslock) (RDeTable, BCeverad) (notfailed)) reffect (and (BOnTablel iCInlfand) (HandFull) (not ConBlock) (not HandEmpty) (not 30aTable (Cooereed) (not COnBlock) (not HandDmpay) (nct s(niable__Coveeed))
(saction 2ick_13 sparemeters pprecondition land [HnndEmpcy) [ccllock_CCoveedd 1B0n3leck) (not(alled) reffect (end (BIniland) (COnBlock) (HladFa:11) (not B0aBlock) (act HandEmpty) (nct Cemblock_Cooveree (not #Onslock) (not Handimpty) (not conklock__Covereee
(saction #ick_14 rparametera(! :precondition land (HandEmpty) (ConBlock) BinnBlock_Bcoveeede (notfailldd) seffect (end (#CnBlock) icinsand) (8zndFell) (not conBlock) (not. Hondampty) (not sonRlock Bonklookk_CCvvele #Covered) (not conklook) (not HandRmpry)
SDoBlock_DCovered))
(saction Put_15 sparemeters sprecondition land (HandFull) (AInHand) (notfailed))
seffect (and (AOnTable) (ffandEmptyl Inot AInfland) (not HandFull) (not AInfland)
HandFull)
(saction Put_16 spareneters sprecondition land (HandFul (RTaHand) (not
reffect fand (BOnTable) (llandEmptyl fnot BandFulli Ino: BInfland) (not HandFull) }InHand))
(saction sut_17 sparaneters sprecondition (HandFull) (CTalland) (notrailed))
reffect (ead (COnTable) (llandEmpty) fnot BandFull1 tnos CInfland) (not HandFulll (nct CinHand))
(saction Stack_18 sparsmeters gprecondition IllanFulll) (CIalland) (BOnTablet (nostalled))
reffect (end (BOnTable_3Covest (COnBlock) (HandEmpty) taoz HandFulll (nct sUntable) cinkand) Handrull) (not sonTable) (net CTnHand))
(saction Stack_19 sparemeteis() sprecondition land (HandFull) (C0mBlock) (Binlannd las:failed)) (end (8CnBlock) iconslock_ (ccoveerd)) (HandEmptyl Inot HandFull] (not conB lock) (not BTnRand) (not RandPul (not conkliock) (not 3Ialland))
(saction Stack_20 :paremeters(1
:precondition land (HandFull) (A0nRlock) (BTnRond) (nosfailed) reffect (and (BOnBlock) Bo0nBoook_Aoveeree (HandEmpty) Ino: BandPoll) (not 3Ialland) (not. AOnBlock) (not MandPPlll (not BInlland: (net sinblock)
14

---

Published as a conference paper at ICLR 2022
(saction stack_21 sparsmeters()
:precondition land HandFull) (AInfland (BOnTablet inostalled))
reffect (and [BonIable_3Ceveeee) (AOnBlock) (HendEmpty) lnos AInfland) (nct aendkull) (act sunrable) (not AinHand) (not Handrull) (not scntable)]
(saction stack_22 :paremetera(1
:precendition land IllandFull) (Cinlland) (BOnBlocki (ns:Seiled)
reffect (end (#cnBlock_stevezed) (c0nelock) (kandkmptyl Inoz HandFull) (not sCnRlock) CTnRand) #andFull) (not Bonālock) (not CInHand))
(raction Stack_23 sparemeters
sprecendition land (HandFull) (Contablel (8Insandi Inosfailed)) seffect (and (RCnRlock) (not BIalland) (aot ConTable) (not flanPPllly (not BInlland: Contable);
(HandRmpty) HandFull)
(saction Stack_24 sparsmeters gprecondition IllanFull) (ATalland) (ConBlock: (nostailed)
reffect (ead [Dnblocc_CCeveeee (A0nBlock) (landEmpty) laoz AIniland) (nct sendrull) cUmblock) (not AinHand) (not Handrull) (not COmBlock)
(saction stack_25 :garemeteis(1
:precondition land (HlndFulll (ADntable) (CInliand: (ns:Sailed)
reffect (cenBlock) 140ntable_ ACooeredd (sandimpty) Inot Handrull) (net #Ontable) (nct CInlland) (not HandPull) (not AOnTable) (not Cintand))
(saction Stack_26 sparemeters :precendition (HandFull) (AInHand) (contablat (nosfoiled))
reffect (end (COnieble_CCevezes) (AOnBlock) (HandEmpty) Inos Alnlaan) (not sondPull) (not cintable) (not AInHand) (not HandFull) (not contable)
dsaction Stack_27 sparameters
sprecendition land (Handrull) (AomBlock) (cinsendi inssfailed)) seffect (and (CCnBlock) iA0nBlock ACovered) [aovslcck__Aoveree)) (HondRmptyl Inoz HandFull) (not ACmBlock) CInlland) HandFull) (not 40nBloox)
(not CInlland))
tsaction Stack_28 sparemeters
sprecondition land [HandFull) (ADn"ablet (RTnFandi Inoofatleds reffect (and (BOnBlock) (AOnTable ACovered) (MandBmpyy) taos BandFull) (net 3Inlland) (aot AOnTable) (not HandPull) (not BInlland: (nct adniable)]
(saction Stack_29 sparametera sprecendition IllanFulll) (AIalland) (BOnBlock! (no:Ialled))
reffect (end [Jcnelock_acevezeed (A0nBlock) (HandEmptyl Insz AlnHand)
(nct sandF:ll) (not s0nalock) (not AimHand) (not HandFull)
(net 80nB sonBlock)
15

---

Published as a conference paper at ICLR 2022
C LIFTED PDDL DESCRIPTION FOR THE BLOCKS WORLD TASK
contrast, the lifted representation learned below far more compact. Below we provide the learned representation for the domain. Again, we manually rename the predicates and types to help with readability.
(define (donain BlocksWorld) (ssequirements :strips styping) (stypea hand block)
(:predicates
(BleckIniland ?w block)
(sendeul1 2x hand)
(BlockOnBlock block)
(BlcckOnisble ?v block)
(HendEmpty hand]
(RlcckOnTable_Bln Booolle_eoookoeeeee 3w block!
Bewe:onBBocc_BBoocooeeeeee 2w block! (notfailed)
fraction pack-pertitien-
sparemeters (3M - hand ix block)
sprecondition inosfailed) (HandRmpty |8lockonTable 7x)
reffect (and (BlockInHand 7x) (RandFull 7w) (not (SlocxOnTable 7x))
(not (MaadDappy 2w)1)
(saction Pick-partitien-1
sparemeters (3N hand ix block 2y block!
sprecondition land inosfailed] (HancRepty 7wt |RlockonBlock 7X)
BaockonBlock_BLeecCoeee reffect (end (BlsckOnBlock 2y) (BlockIntland 2x) (HandPull ?w)
7y)1
(nct (#lockunslock
(not (Hardempty ?M)
(not Bwwkhnnlook_BBlckkooo Ryt)
tsaction 2ick-partition-10
sparameters (?w hand 7x bleck 2y block1
:precondition land Inosfailed) (dandimpty 2wl [elockontable_secckooo ?x)
[BlockönBlack Py))
seffect (and (RlookTnRand 7y) (BlockanTable 7x) (HandFul1 74)
(nct (#lockcnslock 2y1) (Handampty PM))
(not (RlockonTablee__ e_Blackcovered 7x))
(saction Put-partition-0
sparametera (7M hand block)
pprecondition land inocCalled) (HannFull ?w). (BlockInHand 7x))
reffect (and (RlockCnTable 7x) (HandEmpty Pwl Inst. [BlockInHand 7x))
(not (HandPull 74))]
(saction stack-partition-0
sparametera (7M hand 7x olo 2y block!
sprecondition land InocCailed) (MaanFFl: ?w) (BlockInlland ?x) (BlockOaTable 2y))
Belock@ntable_Blcckeee (net (BaadFull 7w)) (not (BlockOnTTable 7y1)
2y) (BleckonBlock 2x) (HandEmpty 2w)
(not (BlockInlland (x)))
(saction stack-partition-i
sparameters (?x hand 2x bleck 2y block! :precondition inotfailed) (Handrul2 ?w) (elockonslock (blockinHand 2y)
reffect (and (BlockOnBlock ?yI BrockonBlock_Bleccoooo ?x) (BandEapty ?w)
(nct (HandPull 2w)) (not (BlockO:Block ?x!)
(not (BlockInnaand ?))
16

---

Published as a conference paper at ICLR 2022
A task might then be specified follows:
(problem atack)
(:domain BlocksWorld)
(sobjacta hand Hond
A
(sinit: (RlockanTable A)
(BlockOsTable B)
(BlockO:Table C)
(Handimpty hand) (notifal lod)
(:goal (and (Block0nBlock
#ilockOnBlock Boo__kkkccCoeereed CI
(BlockOaTable Blockkoceeeeeoo
D ILLUSTRATION OF LEARNED OPERATORS
In Figure 6 we illustrate one propositional operator for picking block B off block C, while Figure illustrates the lifted version of the same operator for picking any block X off any block Y.
(caction Pick-partition-10 :parametera ()
:precondition (and (aymbol_10)
A
(symbol_15) (symbol_6))
reffect (and (symbol_3) 3) (aymbol_4)
(a) symbol.10
(b) symbol.15
(c) symbo1..6
(symbel_1)
symbol_6)
(not symbol_10)
(not symbo [51)
(g) Propositional PDDL operator for one of the Pick option partitions.
(d) symbol.3
(e) symbol_4
(f) symbo 1_1
Figure The learned propositional operator for a Piick ck action describing picking off C. execute the action, the hand must be empty (symbol_1) 1_10), C must be the table and covered by block (symbo1.15), and B must be on top block and uncovered (symbol.6). After execution, is in the hand (symbol 3), C is on the table and clear (symbol.44 14), and the hand is full (symbol 1)) We visualise each propositional symbol by sampling from it, and randomly sampling the remaining independent state variables (since each symbol is distribution transparency is due to the averaging over the independent state variables. Note that we must learn one operator for every pair of blocks.
subset of state variables). The
17

---

Published as a conference paper at ICLR 2022
(saction #ick-partition-10
sporametera 17w typea typel
7y typel)
:precondition
(aymbol_4 ?w)
(a) symbol_4
(b) symbol_5
(c) symbol_2
(symbel_5 ?xt (aymbo1_2 ?y)
(symbol_ 7y1 taymaol_ 7x) (aymbol_ 2w) (not, (aymbol_2 (not (symbol_4 2x0) (not (symbol_s 2x1))
(d) symbol_0
(c) symbol.3
(f) synbol.1
(g) Lifted PDDL operator for Pick action.
The learned lifted operator for a Pick action describing picking a block off another, In order pick up block Y, it must be block which itself is the table, and the hand must be empty. As a result, the hand is not empty, Y is now in the hand, and X is on the table and clear. type0 refers the "hand" type, while type1 refers to the "block" type.
Figure 7:
E MINECRAFT TASK DETAILS
Our Minecraft tasks are procedurally generated, consisting of five rooms with various items positioned throughout. Rooms are connected with either regular doors which can opened by direct interaction, or puzzle doors which require the agent to pull lever to open. The world is described by the state of each of the objects (given directly by each object's appearance as a 600 x 800 RGB image), the agent's 's view, and current inventory. Figure illustrates the state of each object in the world at the beginning one of the tasks.
Figure 8: The state of each object in the world at the start of the task. From left to right, the images represent the agent's point of view, the four doors, the pickaxe, the chest, and the redstone and gold blocks. The inventory not shown here.
The agent is provided with the following high-level skills:
(i) WalkIoItem the agent will approach an item if is the same room.
(ii) #ttackBlockk- agent will break block, provided it is near the block and holding the pickaxe.
(iii) PickupItem-the the agent will collect the item if it is standing in front of
(iv) WalkToNorthDoor the agent will approach the northern door in the current room.
(v) WalkToSoouthDoor-thhh agent will approach the southern door in the current room. (vi) WaikThhooughooor-tt the agent will walk through a door to the next room, provided the door is open.
(vii) Cra the agent will create a new item from ingredients its inventory, provided it near the crafting table.
(viii) OpenChest-Bee the agent will open the chest, provided it is standing in front of it and possesses the clock.
(ix) ToogleDoors agent will open close the door directly in front of it.
Execution is stochastic- -peeaastccooeennngggg doors occasionally fails, and the navigation skills are noisy their execution.
18

---

Published as a conference paper at ICLR 2022
F
CRAFTING TASK
The Crafting domain consists of a 12 x 12 grid containing the following objects:
(i) gold, surrounded by water which only be accessed once bridge is laid;
(ii) iron objects;
(ii) 4 grass objects;
(iv) 4 wood objects;
(v) 4 rock objects; and
(vi) 3 workshops.
Furthermore, a stick can be crafted if the agent has picked up a wood object and is near the second workshop. while bridge can be crafted if the agent is near the third workshop and possesses wood and iron. Finally, the agent can complete the task by crafting a gold arrow, which requires it to be near the first workshop and holding gold and stick.
The state of each object characterised by whether it is on the grid (present), has been picked up by the agent (pi (picked) does not yet exist (non-exiss existent); the state of the world is given by the state of each object, as well as the agent's egocentric view. The agent is given four skills: WalkTo, P Pickup, Place and Craft-an object to it, and any particular item can only be crafted if the agent has picked up the correct "ingredients". Furthermore, the gold object is separated by water and can only be accessed once the agent has placed a bridge, which must first be crafted. To solve the task, the agent is required to construct a gold arrow, which involves collecting wood and iron create bridge, then collecting the gold, crafting stick out of wood, and then finally crafting the arrow using the stick and gold.
only be picked up when the agent is adjacent
G LEARNING A PORTABLE REPRESENTATION FOR MINECRAFT
In this section, we describe the exact details for learning a representation of Minecraft task. Pseudocode for the approach (independent of the domain) is provided in Appendix[[
In order to learn a high-level representation, we first apply a series of preprocessing steps to reduce the dimensionality of the state space. We downscale images to 160 x 120 and then convert the resulting images to greyscale. We apply principal component analysis (Pearson) 1901) batch of images collected from the different tasks and keep the top 40 principal components. This allows us to represent each object (except the inventory, which a one-hot encoded vector of length 5) as a vector of length 40.
Partitioning We collect data from task by executing options uniformly at random. We record state transition data, as well as which options could be executed at each state. We then partition options using the DBSCAN clustering algorithm (Ester al. 1996) cluster the terminating states of each option into separate effects. This approximately preserves the subgoal property as described in Section 2 and previous work (Andersen & Konidaris 2017 Konidaris et al.) 2018)) Ames al.) 2018). For each pair of partitioned options, we check whether there is significant overlap in their initiating states (again using DBSCAN), If the initiating states overlap significantly, the partitions are merged to account for probabilistic effects.
Preconditions Next, the agent learns precondition classifier for each of these approximately partitioned options using an SVM (Cortes & Vapnik, 1995) with Platt scaling (Platt 1999). We use states initially collected as negative examples, and data from the actual transitions as positive examples. We employ a simple feature selection procedure to determine which objects are relevant to the option's precondition. We first compute the accuracy of the SVM applied the object the option operates on, performing a grid search to find the best hyperparameters for the SVM using f-lold cross validation. Then, for every other object in the environment, when that object's features are added to the SVM. Any object that increases the SVM accuracy is kept. Pseudocode for this procedure is outlined in Figure 9
compute the SVM's accuracy
19

---

Published as a conference paper at ICLR 2022
Having determined the relevant objects, fit probabilistic SVM to the relevant objects' data. Note that we learn a single SVM for a given precondition. Thus if the precondition includes two objects, then the SVM will learn classifier over both objects' features jointly.
1: procedure FEATURESFLEECTIN
2: Given: affected objects Mask, positive start states p. negative start states n, set of objects
3: Fit classifier over only objects in the mask
4: classifier +- FITCLASSIFIER (start, negative, mask)
5: initScore &- classifier.scom
6: Keep +
7:
for each object E M Mask do classifier + ASSSIIRSSR (starr negative, Mask U {object))
8:
9:
newscore classifier.scom if newscore > initScore then Keep the object if improves the score
10:
12: end
Keep < Keep U {object]
14: end for
return Mask U Keep
16:
end procedure
Figure 9: Pseudocode for simple feature selection procedure.
Effects A kernel density estimator (KDE) (Rosenblati, 1956) with Gaussian kernel is used to estimate the effect each partitioned option. We learn distributions over only the objects affected by the option, learning one KDE for each object. We use grid search with 3-fold cross validation to find the best bandwidth hyperparameter for each estimator. We fit a single KDE to each object separately, since the state space has already been factored into these objects. Each of these KDEs is an abstract symbol in our propositional PDDL representation.
Propositional PDDL For each partitioned option. now have classifier and set of effect distributions (propositions). However, to generate the PDDL, the precondition must be specified in terms of these propositions. We use the
approach as Konidaris al. (2018) to generate the PDDL: for all combinations of valid effect distributions, we test whether data sampled from their conjunction is evaluated positively by our classifiers. If they are, then that combination of distributions serves as the precondition of the high-level operator. This procedure is described in Figure I0
Type Inference To determine the type of each object, we first assume that they all belong to their own type. For each object, we compute its effect profile by extracting the effect propositions that occur under each option. Figure illustrates this process.
For each pair objects, we then determine whether the effect profiles are similar. This task made easier because certain objects do not undergo effects with certain options. For example, the gold block cannot be toggled, while a door can. Thus it easy to see that they are not of the same type. To determine whether two distributions than a certain threshold. Having determined the types, we can simply replace all similar propositions with predicate parameterised by an object of that type, described by Figure 12
similar, simply check whether the KL-divergence is less
20

---

Published as a conference paper at ICLR 2022
1: procedure BUILDPPDDLOPERATOR
2: Given: precondition classifier classifier, current effect effect, all effects Bffects
3: Operators +-
4: Symbols t- 0
5:
for each condidate E (PEffects) do samples t SAMPLE( condidate)
For all possible effect combinations Sample from the distributions Query the classifier with the data
6:
7:
prob PREDICT(classiiier sample) if prob > 0 then
8:
9:
prob then
10:
& Construct the new operator with the existing effects operator t- {candidate, effect)
12:
else
13:
b Add a probabilistic failure case
newEffect fail, 1, with probability (1 - pprob)
14:
effect, with probability prob
15:
operalor + candidale, new
end if
17:
Operators + Operators U {operator]
Symbols < Symbols U (candidate) U {effect) end if
19:
20:
end for
21:
return Operators, Symbols
22:
end procedure
Figure 10: Pseudocode for constructing propositional PPDDL operators.
1: procedure COMPUTEEFFECTS
2: Given: object i, option o, PPDDL operators Operalors
3: Get only the operators that model option
4: Operators +- {operator Voperator Operators, REFERSToo(Operator o)}
5: Effects +- 0
6:
for each {,, effect) E Operators do Extract the effect propositions that refer distributions object i
7:
8: OperatorEfffct + {prop Vprop E effect, REFERSTO(prop, i)}
9: Effects Efffects [Operator(ffect)
10: end for
return Effects
12:
end procedure
Figure 1I: Pscudocode for computing the effect distributions under an option for a given object.
Problem-Specific Instantiation Finally, we again use DBSCAN to partition our subgoal options, but this time using problem-specific state variables. Each of these clusters is then added our representation problem-specific proposition. To ground the operators, we add the start and end clusters (problem-specific propositions) to the precondition and effects the PPDDL operator. We also record the grounded object that appears in the parameter list of each operator, and add precondition predicate (fluent) to ensure that only those particular objects can be modified. Without this final step, the agent would, for example, believe it can open any door while standing in front of door at a particular location. We have thus linked the particular door to a particular location in the domain.
21

---

Efffett F.ffects J (estimator) Published as a conference paper at ICLR 2022
1: procedure MERGE
2: Given: objects M. type T, PPDDL operators Operators, propositions Propositions
3: Find the first object matching the type
4: archetype 0
5:
for each object E M do if ISTYPE( object,, T) then archelype object break end
6:
7:
8:
9:
10:
end for
Remove propositions with objects of type T that are not the archetype
12:
Removed < {prop Vprop E Propositions, IST T), -REFERSTO(prup archetype)}
13:
Keep operators that do not contain the removed propositions Operators + (op Vop E Operators, Removed op = 0}
14:
15:
return Operators, Propositions \ Removed
16:
end procedure
Figure 12: Pseudocode for lifting propositions to typed predicates.
H LEARNING REPRESENTATION FOR THE CRAFTING DOMAIN
To learn representation for the Crafting domain, use the exact procedures described in the previous section. However, we use the following hyperparameters:
ii) Partitioning was achieved using DBSCAN with 0.1.
Preconditions were learned using an SVM with C 4.
(iii) The kernel density estimators used for the effects had bandwidth parameter 0.001.
I PSEUDOCODE
Below we present pseudocode describing our approach building a typed, object-centric PPDDL representation for an arbitrary domain. Some subroutines used in the pseudocode below are outlined in the previous section.
1: procedure
2: Given: T' state-option transitions {(s,,,,,,,,,,,,, Fy 01, s/, S i S T), set of objects M
3: Partition options into subgoal options
4: SubgoalOptions t
5:
for each E do
6:
Set of initial states for option Set terminating states for option for all K C such that Pr(s' 0) Pr(s' 1s,,0)ss, I, E 3 do
I -- {8 (8, 0,,,00,,3 D) + {s' (...0,,' s . E D}
7:
8:
9:
P+ (o, K, K Vs E K, (8,,0,8 8', D)) & Start and end states for a partition SubgoalOptions
10:
SubgoolOption U (P)
11:
end for end for
13:
Estimate preconditions and effects Preconditions, Effects + 0 for each (., start, end) E SubgonlOptions do mask f- (AKPPUTTMAAK((Satrt end)
14:
16:
> List the objects that change state
17:
negative + S start
18:
features +- TEATURRSSLLECTTONNMnA,,,, start, negative)
19:
classifier + FITCLASSIFER start, negative, features)
20:
Preconditions t- Preconditions U cclassifier)
21:
estimator + FITESTIMATOR mask, end) b Fit over only objects that change
22:
22

---

Published as a conference paper at ICLR 2022
23:
end for
24:
Build propositional PPDDL
25:
Operators, Propositions t-
26:
for each precondation, effect E (Preconditions Effects) do
27:
op, symbols + BULLLPPDDOOPRRTOOR oprropDDDD,,EAAAoooootot,,,,,,,, (precondition, effect, Effects) Operators < Operators U {op}
28:
29:
Propositions + Propositions U symbols
end for
31:
Infer object types
32:
PffProfile <- 0
33:
for each object m
34:
for each E O do EffProfilemm, o) < COMPUTEERVECTS(( o, Operators)
35:
36:
end for end for
37:
38:
Types (K) (.)Profile(m;;))) FiPrrofilemm E O, m, my KMKCM
39:
& Generate typed PPDDL
40:
TypedOperaaors. Predicates <- for each type Types do D Replace propositions and operators over objects of same type with lifted versions ops, predicates < MERGE(M, type, Operators, Proposilions) TypeatOssators +- TypedOOperators U ops
42:
43:
44:
45:
Predicates + Predicates U predicates
46:
end for
47:
Instantiate typed PPDDL new task
48:
for each {o, start, end) E SubgoalOptions do
49:
Lx 4- {ar Vs start, s' end, (8, x., 0, s) ED Bx < {a' Vs E start, s' E -endd E Ix. (s. x'))'' D} for all Lx such that Pr(æ' Ii, Pr(x' Zj 2,,0 < fx' Vs E start, s' E end, x E K. (8, 0, E E Predicates +- Predicates U (A) mask +- COMPUTEMASK > Link ppecifii symbols in precondition and effect to the affected objects OypeadOperttors +- GROUND ThypedOperators X, mask)
50:
Lx, x' 3x do
52:
& Add mroblem-specifie symbols
54:
end)
> Computes the affected objects
55:
56:
57:
end for end for
58:
59:
return TypedOperators, Predicates
60:
end procedure
23

---

Published as a conference paper at ICLR 2022
J ADDITIONAL RESULTS FOR THE CRAFTING DOMAIN
Based their effects, each object grouped into of the six types below:
Type
Objects
Remark
type0 agent
The agent
typel
gold0, iron2, iron3, wood0, woodl, wood2, wood3 iron0, ironl, grasso, grassl, grass2, grass3, rock0, rockl, rock2, rock3 workshop0, workshopl, workshop2 stick, goldarrow
Objects that were picked up and used craft new items Objects that were picked up but never used crafting
type2
type3
Objects that cannot be modified
type4
Objects that can be created but not picked up Objects that can be placed
type5
bridge
Table 2: The various objects in the domain along with their learned type.
symbol 27
symbol 51
symbol 25
symbol
symbol 25
symbol 30
symbol 21
(saction craft-poattiiion---1kk..
sparametera !7a typeo 7b typel 2a - type4 ?d type4)
:precondition (aymbol_27
taymbol_51
taymbo1_25
(aymbol_ 21 rd) [= rid 70) 0) (id 2at 1) (id 26) 21)
11d 7d) 23)
reffect (and (symbe1_25 2d) (symbo1_21 ?0) (symbo1_30 2b)
(not (symbe1_21 2d)) (not (symbol_25 201) (not (symbol_51 (b)))
A learned typed PDDL operator for the Cra skill, which states that, order to craft gold arrow, the agent must be in front of a particular workshop (symbol.27) and be in possession of the gold (symbol.51) and stick (symbol.25) but not the gold arrow (symbol.21) As a result, the agent finds itself in possession of the gold arrow (symbol 25), but loses the gold (symbol.30) and stick (symbol.21)
Figure 13:
24

---

Published as a conference paper at ICLR 2022
K
VISUALISATION OF MINECRAFT PLAN
21
19
18
WalkTo pickaxe
Pickup pickaxe
WalkNorth door
17
Toggle door
5
WalkThrough door
WalkTo redstone
Attack redstone
25
8
Pickup redstone
15
WalkNorth door
14
26
10
Toggle door
11
WalkThrough door
WalkNorth door
13
13
Toggle door
14
WalkThrough door
WalkNorth door
16
Toggle door
28
17
WalkThrough door
WalkTo gold-block
19
Attack gold-bli block
10
20
Pickup gold-b block
WalkSouth door
8
22
WalkThrough door
23
WalkTo crafting eaalee
24
Craft gold-ingot
25
Craft clock
26
WalkSouth door
WalkThrough door
28
WalkTo chest
29
Open chest
Figure 14: Path traced by the agent executing different options while solving the first task.
25

---

Published as a conference paper at ICLR 2022
L VISUALISING OPERATORS FOR MINECRAFT
Here we illustrate some learned operators for the Minecraft tasks. To see all predicates and operators, please see the following URL: http://sttts.gggl....e.. cwt/viewwwc/ewwweeooooo........c mine pddl
(saction Open-Chest-partitl..e
:pacametera
Lype0
Lype6
Py type9)
:precondition
(sotfailed)
(aymbol_13 7w) (aymbol _A 7x)
(symbol_55 Py] (psymbol_ 61)
(a)
(b) symbol.4
(c)
seffact (and (aymbol
(aymbol_S)
symbol.13
symbol_55
(not (symbol_4 7x))
(symbol 2w)
(g) A learned typed PDDL operator for the Open-Chest: skill. The predicate underlined in red indicates problem-specific symbol that must be relearned for each new task, while the rest the operator can be safely transferred.
(d)
(c)
(f
psymbol_8
symbol.58
symbol.59
Figure 15: Our approach learns that, order to open a chest, the agent must be standing front of chest (symbol_13) the chest must be closed (symbol_4), the inventory must contain clock (symbol_59) and the agent must be standing a certain location (psymbol_8) The result is that the agent finds itself in front of an open chest (symbol 58) and the chest is open (s ysymbol 59). tppe0 refers to the "agent" type, type 6 the "chest" type and type the "inventory" type.
(raction Walk partition :perameters [2w type0;
sprecendition (and (notfcilad) (aymbol_46 ?w)
(psymbol_00)
reffect (symbol_ 7M) (paymbol_1)
(b)
(not: (symbol 46 74) (not [psymbo) 311)
symbol.46
psymbol.0
(e) Typed PDDL operator
partition Palk-To option. The predicate underlined in red indi- cates problem-specific symbol that must be relearned for each new task, while the rest of the operator can be safely transferred.
(c)
(d)
symbo1.11
psymboll1
Figure 16: Abstract operator that models the agent walking to the crafting table. order to do so, the agent must be standing the middle of room (symbol-46 particular location (psymbol.0) As result, the agent finds itself in front of the crafting table (symbol.1) at a particular location (psymbol 1).
26

---

Published as a conference paper at ICLR 2022
Isaction throughh-Door-porttttt----------
:parameters (2w type0 2x typel)
:presandition
(notfailed)
(symbo 1 _38 7M) (symbol 1464 ?x)
(- (id 2x) 1psymbol 241)
(a)
(b) symbol 64
reffect (and (symbol_50 2w)
(not (symbol 25r) [psymbot__22 12)
symbol
paymbol 24
(paymbo 2212))
(1) Typed PDDL operator for a partition the Throught Through-Deor option. The predicate under- lined in red indicates problem-speciee speciifi symbol that must be relearned for each new task, while the rest of operator can safely transferred.
(d)
(e)
symbol _50
psymbol 1ammmoll_11
Figure 17: Abstract operator that models the agent walking through a door. In order to do so, the agent must be standing in front of an open door (symbol 1_38) particular location (psymbol. __24), and the door must be open (symbola 1_64). As a result, the agent finds itself in the middle of a room (symbol_50 at particular location (psymbol.12).
lraction Attack-partisicn----1- :parametera type0 2x type?)
:precordition land (nottailed]
(symbol_15 ?w) (symbol_2 Px)
(paymbol 1711 :effect (and (symbol_19 19 2x) (symbol_20 2w)
(a)
(b) symbol_2
(c)
(not (symbol 2x)
(not (symbol_15 2)))
symbo1.15
psymbol.1 _17
(f) Typed PDDL operator for partition of the Attack option. The predicate underlined red indicates problem-specific symbol that must be relearned for each new task, while the rest of the operator can be safely transferred.
(d)
symbo1 1
symbe1 20
Figure 18: Abstract operator that models the agent attacking an object. In order to do so, the agent must be standing in front of gold block (symbol 1.5) at particular location (psymbo 17), and the gold block must be whole (symbol_2). As result, the agent finds itself in front of disintegrated block (symbol_20) and the gold block is disintegrated (symbol_19)
27

---

Published as a conference paper at ICLR 2022
M EXAMPLES OF FAILURE CASES
Below are some examples of errors that occur when constructing our abstract representation. Since there several phases involving clustering, classification and density estimation, expect various learning errors to occur throughout. These errors could have numerous causes, such as insufficient data or suboptimal hyperparameters.
M.1 PARTITIONING ERRORS
(a) Set of start states for one partition of the
(b) Set of end states for one partition of the Attaack option.
At Btackk option.
(c) Set of start states for another partition of the Attack tack option.
(d) Set of end states for another partition the At taak option.
Figure 19:
above example, the partitioning procedure has generated two partitioned options for breaking the gold block, where there should only be one. They are functionally equivalent, but because the strange shadows on the left the image patch and the subsequent PCA representation, the clustering algorithm has produced one extra partition.
(a) Set of states one partition of the ToggleDoor option.
(b) end states one partition of the ToggleDoog option.
(c) Set of states for another partition of the loogl eloor option.
(d) Set of end for another partition of the Toggl elloor option.
Figure 20: In this example, the partitioning has clustered noisy samples into additional partition of the Togg eDoor option. While the top row shows the case where the state of the door changes from open closed, the bottom row is relatively useless noisy operator. We will subsequently learn a precondition and effect for this partition, but likely will not used by the planner.
28

---

Published as a conference paper at ICLR 2022
M.2 PRECONDITION/FEATURE SELECTION ERRORS
(a) precondition attacking gold block. The top image represents the agent's 's view (in front of the block), while the bottom image is the state of the block (unbroken).
(b) The precondition for walking to a closed door. The top image represents the agent's 's view (in room), while the bottom image is the state of the door (closed) and the state of the inventory.
Figure 21: In the left example, the classifier predicts that the gold block can be broken when the agent in front of it. However, this is not quite correct, since the agent must also have the pickaxe to break the block. In this case, the issue occurs because the data only included states where the agent reached the gold block with the pickaxe. Therefore, the agent did not observe states where it was front of the block without the pickaxe, and thus concluded that the pickaxe is irrelevant to the precondition. In the right example, the classifier has overfitted to the data and predicts that the agent can only walk the door when it has the pickaxe.
M.3 PPDDL CONSTRUCTION ERROR
The quality of the PPDDL operators depends on how accurately the precondition classifiers and effect estimators are learned. Any error in learning result in imperfect PPDDL operators, seen below.
liaction Craft-partsttion-1-222
:parameters (?M typea 2x type
:presardition (and (notfailed)
(aymbo1_56 (symbol_29 7x)
[psymbol_111
reffect (probabilistic, 0.21
(a)
(b)
(c)
(not (notfailed)
symbol.56
symbol_29
psymbol.1
0.79
(and (symbol __ 7x) (aymbol_11 7w)
(not (symbol_23 _29 2x)
(not (aymbol
(1) Typed PPDDL operator for a partition of the Cr a ft. option.
(d)
(e)
symbo1.57
symbol_11
Figure 22: Abstract operator that models the agent crafting gold ingot. In order to do so, the agent must be standing front of the crafting table (symbol.3 1_56) particular location (psymbol 1 1_1), and must have the gold block in its inventory (symbol. 1.291_9) As result, the agent finds itself in front of the crafting table (symbol.11 is Anterministicchhweeee however, due estimation enrors, the PPDDL operator predicts that it will only succeed with probability 0.79.
with a gold ingot in its inventory (symbol.5 This option
29

---

Published as a conference paper at ICLR 2022
M.4 TYPE INFERENCE ERROR
We observe that occasionally the procedure will not discover the correct types. In the example below, instead of discovering single type for all four doors, our approach predicts that one door is different from the others and is placed in its own class
Type
Name
Object(s)
0
Agent
0
1
Pickaxe
1
2
Door1
2, 3, 4
3
Door2
5
4
Redstone Block
6
5
Gold Block
7
6
Chest
8
7
Inventory
9
Table 3: grouping of objects into types. Note that one of the doors is allocated its own type.
30


Reviews:
[{"review_id": "sOjslt2UsB_", "reviewer": "Reviewer_nhv5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The focus of this paper is on learning representations of an environment that allow for planning, which is an important problem. The main driver for the presented improvements is the assumption that the world can be organized according to objects that are relevant for many different tasks. Recently, there has been an increasing interest in incorporating such real-world priors in representation learning methods, which makes the overall focus highly relevant. The paper is also well written and the ideas are presented clearly. Further, the paper does a good job at crediting prior work wherever appropriate.\n\nThe main improvements in this paper compared to prior work of Konidaris et al. (2018) are\n\n1. A method for estimating object types by evaluating their effect distributions when applying (subgoal) options. \n\n2. A method for integrating task-specific information in the learned representations, i.e. grounding them for particular problem instantiations.\n\nI think (1) is quite clever since it allows for a grouping into object types that focuses on their functionality, which transcendent a grouping based on other features such as appearance. On the other hand, it is not quite clear to me why a grouping based on effect distributions alone is preferred over one that also considers the pre-condition? The assumption that an option only affects the object it acts upon is quite strong and in my view somewhat unrealistic, yet it wasn't quite clear to me how this relates to this choice either.\n\nRegarding (2), I find that the current presentation falls somewhat short. For example, it is unclear to me how it is determined what objects/operators require grounding (is this prior knowledge that is assumed given. Eg. for the workbench or door in the experiments). It is also not clear to me how the clustering proceeds for X. In particular, is X still partitioned according to objects similar to S? And how is the result from clustering X connected to clustering S, i.e. how are the clusters mapped onto each other and how is it ensured that a similar split is obtained? I am particularly interested in understanding how much domain knowledge is injected in (2). It may very well be that I misunderstood something here, so further clarification is much appreciated.\n\nThe approach is evaluated on three different environments. On Blocks World it is demonstrated how the 'correct' environment representation is learned and the benefits from merging into object types is clearly demonstrated.\n\nOn the Craft environment from Andreas et al. (2017) it is shown how the approach scales to situations where there are many different objects, and further how grounding can be achieved for operators that require identities for a specific problem instantiation. Here the identities are derived from the option masks, and it is shown how a grounded operator is learned. However, similar to above it is unclear to me what are the steps needed to achieve this result, i.e. how is the need for having identities identified (or is it assumed given?), how is it determined which objects require ids (or is it assumed given?), and how is the id connected to the object. What I liked about this experiment is how not the 'ground truth' types are recovered but rather a suboptimal grouping based on the data collected so far. It also suggests that it may be worthwhile to relearn the object types after having interacted with the environment for a number of steps, which is something that could be commented on.\n\nOn the MineCraft environment from Johnson et al. (2016) it is shown how this approach can be combined when learning feature representations of objects from their pixel representation rather than using pre-specified features. Here each object's appearance is given by a high-dimensional image although it is still assumed that objects are pre-segmented. Here, the learned operators must be grounded to consider a particular door (and I have similar questions as above). By considering 5 procedurally generated tasks it is also demonstrated how object types allow for transfer compared to prior work of Konidaris et al. (2018), which clearly demonstrates the benefits of this approach.\n\nAs a final comment, I would have liked to see some more discussion of the limitations of this approach. The appendix includes several failure cases, which are interesting, but the framework presented here makes several strong assumptions that are worth commenting on: (1) the frame assumption, which appears crucial for learning effect distributions, (2) that options only affect the object they act upon, (3) that the world comes pre-segmented according to objects. When also inferring the objects themselves (i.e. the segregation process) the merging into object types may affect one another, since by adopting a different notion of an object their effect distribution is likely to change.\n", "summary_of_the_paper": "This paper proposes a method for learning an object-centric symbolic representation of an environment that allows for planning. It extends the framework introduced in Konidaris et al. (2018), which learns a symbolic representation of an environment that can be expressed using a PDDL for planning. Importantly, the method presented in Konidaris et al. (2018) does not make assumptions about structure in the environment and how different states may relate to one another. As a consequence, the learned representations are highly tasks-specific and transfer/generalization to new tasks and across states is not possible. \n\nIn this paper it is assumed that the world consists of objects and that similar objects are common across tasks. This enables object-centric representations (and the associated model) to be reused across tasks, and organizing objects according to 'object types' that behave similarly when acted upon. This is expected to improve learning efficiency and generalization/transfer to new tasks. \n\nThe method presented here incorporates techniques for merging objects into object types, and integrating problem-specific information to ground these representations into specific tasks. The working of these techniques is validated experimentally, and it is shown how greater sample efficiency can be obtained when generalizing to new tasks.", "summary_of_the_review": "This is a well-written paper that introduces several techniques into an existing framework for learning symbolic representations for planning. The proposed contributions are interesting and the improvement over prior work is clearly demonstrated. \n\nWhile the final approach itself is still limited, in the sense that it relies on several strong assumptions, it advances what symbolic approaches are currently capable of. Further, by highlighting the trade-off between creating object types and problem-specific grounding it contributes an interesting problem that may not yet be on everyone's radar.\n\nWhile I would like to see this paper accepted I also think it can still be improved. The presented approach is quite complex, and although the authors do a good job overall, the presentation for the grounding is lacking and it is not made sufficiently clear how this is achieved and what knowledge is assumed to arrive at the presented results. I also suggest that the authors consider creating a figure to accompany figure 1 that specifically visualizes step 1 (and step 5). In order to make the paper accessible to a wider audience, an introduction to PPDDL notation would be helpful to include. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J__Q6EyuiiV", "reviewer": "Reviewer_PYjf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is overall well-written, clearly presented and generally of high quality. The problem of learning sample-efficient abstractions for long-term planning is an important and difficult problem, and taking an object-centric approach to address this problem is very timely and of high significance. The proposed method is largely based on a framework introduced in prior work (Konidaris et al., 2018), but introduces sufficient novelty in terms of integrating object-centricity in an elegant way into this framework. The experimental evaluation is carried out on sufficiently complex environments to demonstrate that the method can solve tasks that are not trivial.\n\nMy main concern with this paper is with regards to its sample complexity claims and limited experimental comparison. The paper claims (in the abstract) that the proposed method results in being able to obtain a successful agent for long-term planning „with considerably fewer environment interactions“. Given this central statement, I would expect some form of quantitative evaluation against a baseline that demonstrates this strong reduction in environment interactions (which unfortunately is not provided). Potentially it is trivial to see that this method uses fewer environment interactions than prior works (for example the works by Kaiser et al. (2020) and Hafner et al. (2021), as discussed in the paper) — this would not be a fair comparison, however, as the presented method assumes that environment observations are pre-factored and pre-processed into object components (e.g. provided as abstract per-object feature vector or as cropped images around individual objects). Extracting object information in this way without direct supervision is a highly non-trivial task, which might partially explain the sample inefficiency of earlier methods (Kaiser et al., 2020 and Hafner et al., 2021). This present paper sidesteps this issue by providing ground truth information. I would recommend either 1) significantly reducing the prominence of the sample efficiency claims in the paper or 2) providing a fair experimental comparison against a baseline that has access to the same factored, pre-processed observations and abstract options (instead of low-level actions) as the presented technique.\n", "summary_of_the_paper": "This paper introduces a method for learning symbolic, object-centric abstractions from object-factored environment observations for long-term planning tasks. It extends the symbolic representation learning framework by Konidaris et al. (2018) by factoring the state into objects, learning object type abstractions, and “lifting” the model to operate on these object-centric, typed abstractions. The method is successfully demonstrated on three environments of complexity ranging from simple per-object discrete feature vectors to a Minecraft long-horizon planning task from (object-factored) pixel observations.", "summary_of_the_review": "Despite the above-mentioned limitations, I think that this paper carries sufficient novelty and is an interesting, significant extension of prior work in this area, which opens up interesting follow-up questions and opportunities for future work. Overall, I recommend “weak accept”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I0IaXEIICxq", "reviewer": "Reviewer_UVNz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is that the author devises a learning framework to abstract object types, pre and post-conditions by grouping object/symbol instances that are option-equivalent or effect-equivalent. Although the proposed method, to some extent, alleviates human efforts to define the planning domain manually, it has two main drawbacks that cannot solve the fundamental problem of the human-defined planning domain. \n\n1. The author claims the learned object-centric abstraction can be transferred to a new task with the same object type. However, it makes no sense if we cannot define how same is w.r.t. the same object type in the new task. For example, we learned the abstraction from picking an apple on the table, and now, given a new task, we want to pick an apple on a tree. These two apples are the same in semantics, but they are different in the planning domain as they have different preconditions for the `pick` action. Such object-centric abstractions are still more of task-specific representation, even for the same objects in semantics. As such, very likely, they have different constraints (preconditions) that limit the actions across different tasks.\n\n2. As mentioned in the paper, we could use Problem-Specific Instantiation tricks to solve the problem of the same-type object but have different preconditions in a different scenario, i.e., adding some task-specific constraints for the object types in different scenarios. However, such a process is exactly the same as how we manually maneuver the planning domain to adapt to a different scenario, and it does not solve the fundamental limitation of the human-defined planning domain: Learned symbolic predicates of specific object types are not general enough to across different task. However, the proposed object-centric abstraction has the same issue. It only groups the similar conditions it observed, but does not actually abstract the preconditions of why an action could be performed. Like the aforementioned picking apple example, if we place the apple on $N$ different heights, the proposed method will lead to $N$ different apple object types as they have N different preconditions. If we place the apple even higher--a condition out of the $N$ previously seen cases, the proposed method still cannot solve this problem as it does not abstract the `reachable` concept for the picking action.  ", "summary_of_the_paper": "This paper proposes a method to automatically learn a PDDL-like abstractions for high-level problem solving based on the different states (either symbolic representation, or high-dimensional embeddings) of the objects in the environment. Such abstractions could support a PDDL solver to find a solution for the task where the abstractions were learned, and the authors claim such abstractions could be shared among different tasks as long as object types remain the same.", "summary_of_the_review": "The authors proposed a learning framework to learn object-centric abstractions for high-level planning, yet it does not solve the fundamental limitation of the manually designed planning domain, and it is more of an autonomous labeling tool that partially alleviates human efforts. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the symbolic representation learning work of Konidaris et al. (2018) to be object-centric, and generalize in this respect. All the reviewers agreed that this is an interesting problem, and that the approach is novel.
Two reviewers gave positive evaluations (6,8), and one reviewer gave a mildly negative review (5), where the main critique is that the method still requires some human effort in designing the planning domain.
While completely alleviating human efforts is definitely a good goal to pursue, I believe that it's too high a bar to ask for in the setting of limited data, and I believe that there are many real world problems where requiring some human effort is not too limiting. 
Therefore I recommend acceptance. 
Please take all reviewer comments into account when preparing the final version.
Decision: Accept (Poster)