Autonomous Learning of Object-Centric Abstractions for High-Level Planning
Authors: Steven James
Benjamin Rosman
George Konidaris
Venue:  (2021)
Keywords: reinforcement learning
planning
PDDL
multitask
transfer
objects

Abstract: We propose a method for autonomously learning an object-centric representation of a continuous and high-dimensional environment that is suitable for planning. Such representations can immediately be transferred between tasks that share the same types of objects, resulting in agents that require fewer samples to learn a model of a new task. We first demonstrate our approach on a simple domain where the agent learns a compact, lifted representation that generalises across objects. We then apply it to a series of Minecraft tasks to learn object-centric representations, including object types—directly from pixel data—that can be leveraged to solve new tasks quickly. The resulting learned representations enable the use of a task-level planner, resulting in an agent capable of forming complex, long-term plans with considerably fewer environment interactions.

Under review as a conference paper at ICLR 2021
AUTONOMOUS LEARNING OF OBJECT-CENTRIC ABSTRACTIONS FOR HIGH-LEVEL PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a method for autonomously learning an object-centrie representation of continuous and sigh-dimensional environment that is suitable for planning. Such representations can immediately be transferred between tasks that share the same types of objects, resulting in agents that require fewer samples learn a model of a new task. We first demonstrate our approach on a simple domain where the agent learns compact, lifted representation that generalises across objects. We then apply it to a series of Minecraft tasks to learn object-centric representations, including object types- sypes-direetly directly from pixel data -that can be leveraged to solve new tasks quickly. The resulting learned representations enable the use of task- level planner, resulting in an agent capable of forming complex, long-term plans with considerably fewer environment interactions I
INTRODUCTION
Model-based methods a promising approach to improving sample efficiency in reinforcement learning. However, they require the agent to either learn highly detailed model-which is infeasible for sufficiently complex problems (Ho et al.) (019)- or build a compact, high-level model that abstracts away unimportant details while retaining only the information required to plan. This raises the question of how best to build such an abstract model. Fortunately, recent work has shown how to learn an abstraction of a task that is provably suitable for planning with a given set of skills (Konidaris) jet al.) 2018). However, these representations are highly task-specific and must be relearned for any new task, or even any small change to an existing task. This makes them fatally impractical, especially for agents that must solve multiple complex tasks.
We extend these methods by incorporating additional structure namelu, that the world consists of objects, and that similar objects are common amongst tasks. This can substantially improve learning efficiency, because object-centric model can be reused wherever that same object appears (within the same task, or across different tasks) and can also be generalised across objects that behave similarly-objec types. We assume that the agent is able to individuate the objects in its environment, and propose a framework for building portable object-centric abstractions given only the data collected by executing high-level skills. These abstractions specify both the abstract object attributes that support high-level planning, and an object-relative lifted transition model that can be instantiated in new task. This reduces the number of samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously seen object types.
We make the following contributions: under the assumption that the agent can individuate objects in its environment, we develop a framework for building portable, object-centric abstractions, and for estimating object types, given only the data collected by executing high-level skills. We also show how to integrate problem-specific information to instantiate these representations in new task. This reduces the samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously objects.
We demonstrate our approach on Blocks World domain, and then apply it to series of Minecraft tasks where agent autonomously learns an abstract representation of a high-dimensional task from raw pixel input. In particular, we use the probabilistic planning domain definition language (PPDDL) (Younes& & Littman 2004) represent learned abstraction, which allows for the use of existing
Mooe results and videos can be found at: httpp://s// goog1 e com//// wwweewmne mi ne-p

---

Under review as a conference paper at ICLR 2021
task-level planners. Our results show that agent leverage these portable abstractions to learn representation of new Minecraft tasks using a diminishing number of samples, allowing it to quickly construct plans consisting of hundreds of low-level actions.
2 BACKGROUND
We assume that tasks are modelled as semi-Markov decision processes M = (,,O,T.R) where (i) S is the state space: (ii) (s)) is the of temporally-extender actions known as options available at state 8; (iii) T describes the transition dynamics, specifying the probability of arriving in state 8' after option is executed from 8; and (iv) R. specifies the reward for reaching state s' after executing option o in state s. An option is defined by the tuple (Lo No: Bo), where Lo is the initiation set that specifies the states which the option can be executed, To is the option policy which specifies the action to execute, and fo specifies the probability of the option terminating execution in each state (Sutton et al. 1999).
We adopt the object-centric formulation from Ugur & Piater (2015): in a task with objects, the state represented by the set (fa, fú, .., [,,,,,,,, ]: vector of features particular to object Note that the feature vector describing each object can itself be arbitrarily complex, such as an image or voxel grid-in in this work we use pixels.
where f, is a vector of the agent's features and f; is
Our state space representation assumes that individual objects have already been factored into their constituent low-level attributes. Practically, this means that the agent is aware that the world consists of objects, but is unaware of what the objects are, or if there are multiple instantiations of the same object present. It is also easy to see that different tasks will likely have differing numbers of objects with potentially arbitrary ordering; any learned abstract representation should be agnostic this.
2.1 STATE ABSTRACTIONS FOR PLANNING
We intend to learn abstract representation suitable for planning. Prior work has shown that sound and complete abstract representation must necessarily be able to estimate the set of initiating and terminating states for each option (Konidaris et al., 2018). In classical planning, this corresponds to the precondition and effect of each high-level action operator (McDermott et al. 1998).
The precondition is defined Pre(o) = Pr(s Io), which is probabilistic classifier that expresses the probability that option o can be executed at state Similarly, the effect or image represents the distribution of states agent may find itself in after executing from states drawn from distribution Z (Konidaris et al. 2018): Im(2. o) = à Js Pr(s' s, s, Z(s) Pr(s E Io)ds, where G == f& (ss) Pr(s E Lo). Since the precondition is a probabilistic classifier and the effect is a probabilistic density estimator, they can be learned directly from option execution data.
We can use preconditions and effects to evaluate the probability of sequence of options plan executing successfully. Given an initial state distribution, the precondition is used evaluate the probability that the first option can execute, and the effects are used to determine the resulting state distribution. We can apply the same logic to the subsequent options to compute the probability the entire plan executing successfully. It follows that these representations are sufficient for evaluating the probability of successfully executing any plan (Konidaris et al. 2018).
Partitioned Options For large or continuous state spaces, estimating Pr(s/ | 8, is difficult because the worst case requires learning a distribution conditioned on every state. However, if we assume that terminating states are independent of starting states, we can make the simplification Pr(s') s, = Pr(s' o). These subgoal options (Precup) (000)) are not overly restrictive, since they refer to options that drive an agent to some set of states with high reliability. Nonetheless, many options are not subgoal. is often possible, however, to partition an option's initiation set into a finite number of subsets, so that it is approximately subgoal when executed from any of the individual subsets. That is, we partition an option o's 's start states into finite regions C such that Pr(s' c) Pr(s'] o, c).c C (Konidaris Konidaris 2017 Konidaris et al. 20188 2018]AAmes Ames et al. 2018). we achieve this in practice by clustering options based their terminating states.
al. 2018). As in prior work (Andersen &
2

---

Under review as a conference paper at ICLR 2021
Factors We adopt the frame assumption, which states that aspects of the world not explicitly affected by an agent's s action remain the same (Pasula et al. 2004). Prior work leverages this to learn factored STRIPS-like (Fikes & Nilsson 1971) representation by computing the option's mask: the state variables explicitly changed by the option (Konidaris et al. 2018). In our formulation, the state space is already factorised into objects, so computing the mask amounts to determining which objects are affected by a given option.
LEARNING PORTABLE CENTRIC REPRESENTATIONS
Although prior work (Konidaris et al. 2018) allows an agent to autonomously learn abstract representation supporting fast task-level planning, that representation lacks generalisability the symbols are distributions over states in the current task, they cannot be reused in new ones. This approach can be fatally expensive in complex domains, where learning an abstract model may be as hard as solving task from scratch, and is therefore pointless if we only want to solve a single task. However, an agent able to reuse aspects its learned representation can amortise the cost of learning many interactions, accelerating learning in later tasks. The key question is what forms of representation support transfer in this way.
One approach is assume that the world consists of objects, and that similar objects are shared across tasks. Approaches like object-oriented MDPs (Diuk et al.] (008) exploit the presence of objects by providing the agent with object-oriented representation, resulting in compact representations that are transferable between tasks sharing the same object classes and dynamics (Guestrin et al. 2003 Diuk al., 2008 Marom & Rosman 2018). Similarly, the classical planning literature has long represented problems in terms of the objects that constitute domain, and operators that can affect their states (McDermott al., 1998). In both cases, however, the question arises to the most appropriate way of building an object-oriented representation of a problem, especially one experienced by the agent at the pixel level. This includes deciding which attributes should be chosen to characterise particular type, as well which objects should belong each type.
We now introduce an object-centric generalisation transfer in tasks when the state space representation consists of features centred on objects in the environment. This is common in robotics, where each object is often isolated from the environment and represented proposed approach in Figure T and the remainder of this section, and provide a detailed pseudocode description in Appendix
learned symbolic representation that admits
point cloud subsequently voxelised occupancy grid. We summarise
Step Par- tition options into subgoal options
Step 2: Generate propositional forward model
Step Lift abstractions based on object type
Step 5: Instantiate ab- stractions for current task
Step 3: Merge objects
into types
Figure I: Learning lifted representations from data. Red nodes represent problem-specific representa- tions, while green nodes abstractions that be transferred between tasks.
3.1 GENERATING A PROPOSITIONAL. MODEL (STEPS 1-2) (AS IN KONIDARIS ET AT.. 2018
The agent begins by executing options using an exploration policy to collect transition data. The first step to partition the options into approximately subgoal options. For each option o and empirical sets of initial and terminating states I, and 3.. the agent partitions I, into subsets K C I, such K,&' E 3.. In practice, this can be achieved by first clustering state transition samples based on terminating states, and then assigning each cluster to partition. Finally, pairs partitions whose initiating states overlap are merged to handle probabilistic effects.
that Pr(s' 0) = Pr(s') 8y, o)s,,
The agent next learns a precondition classifier for each approximately partitioned option. Partitions' initiating states used positive examples, and all other states as negative ones. A feature selection procedure determines which objects are relevant to the precondition. and a classifier is fit using only those objects. A density estimator is then used to estimate the effect distribution for each partitioned
3

---

Under review as a conference paper at ICLR 2021
option. The agent learns distributions over only the objects affected by the option, learning one estimator per object. Together these state distributions form our propositional PPDDL vocabulary V.
For each partitioned option o, the agent has learned a precondition classifier I, and effect estimator 3.. However, to construct PPDDL representto,,,,,,,, both the precondition and effects must be specified in terms of state distributions (propositions) only. Effects are modelled as such, and so pose no problem, but the learned precondition is a classifier rather than state distribution. The agent must therefore iterate through all possible effect distributions to compute whether the skill can be executed there. This achieved by replacing o's precondition classifier with every
(P()) such that Is (s)) > 0, G = -Ippr p. where (?(V) denotes the powerset of V. other words, the agent considers every combination of effect state distributions and draws samples from their conjunction. these samples are classified as positive by lor then the conjunction P is used to represent the precondition. The preconditions and effects are now specified using distributions over state variables, where each distribution is a proposition. We have now learned a PPDDL representation, which is sound and suitable for planning.
3.2 GENERATING LIFTED, TYPED MODEL (STEPS 3-4)
At this point, the agent has learned an abstract, but task-specific, representation. Unfortunately, there no opportunity for transfer (both within the task and between different tasks), because each object is treated as unique. To overcome this, we now propose method for determining object rypes. o(n).
Definition Assume that option o has been partitioned into n subgoal options Object i's profile under option o is denoted by
Profile(i, o) (Prre (2)) ,E, _ ..... (Pre o(n) c0(d))) }],
Eg
where Pre; is the distribution over object i's states present in the precondition for partition k, and Eo(k) is object i's effect distribution
Definition 2. Two objects i and j are option-equivalent if, for given option o, Profile(i, o) Profile(j o). Furthermore, two objects are equivalent if they are option-equivalent for every o in O.
The above definition implies that objects are equivalent if one object can be substituted for another while preserving every operator's abstract preconditions and effects. Such objects can be grouped into the same object type, since they are functionally indistinguishable for the purposes of planning. In practice, however, we can use a weaker condition to construct object types. Since an object-centric skill will usually modify only the object being acted upon, and because have subgoal options that do not depend on the initial state, we can take a similar approach to Ugur & Piater (2015) and group objects by effects only:
Definition 3. Assume that option has been partitioned into subgoal options. Object i's effect profile under option is denoted by
FffectProfile(i, o) = ( Ep(n))
where (o(k) is object i's effect distribution. Two objects i and are effect-equivalent if EffectProfilc(i. = EffectProfile(j.o o) for every o in O.
By computing effect profiles using the propositional representation, the agent can determine whether objects i and j are similar (using an appropriate measure of distribution similarity) and, if so, merge them into the same object type. Propositions representing distributions over individual objects can now be replaced with predicates that are parameterised by types. For example, if there four doors domain, then the agent can replace four propositions representing each door closed with a single ClosedDocr predicate parameterised by object of type door.
3.3 PROBLEM-SPECIRE INSTANTIATION (STEP 5)
If the task dynamics are completely described by the state of each object, as is the case in object- oriented MDPs DDiuk et 2008). then our typed representation is sufficient for planning. However,
"These precondition and effect distributions can be null where appropriate.
4

---

Under review as a conference paper at ICLR 2021
in many domains the object-centric state space is not Markov. For example, in a task where only a particular key opens a particular door, the state of the objects alone is insufficient to describe dynamics-the the identities of the key and door necessary too. A common strategy in this case is to augment an ego- or object-centric state space with problem-specific, allocentric information to preserve the Markov property (Konidaris et al.] 2012] James al., 2020). We denote X' as the space of problem-specife state variables. S remains the original object-centric state space. The above complication does not negate the benefit of learning transferable abstract representations, as our existing operators learned in S can be augmented with propositions over X' on a per-task basis. In general, local information relative to individual objects will transfer between tasks, but problem-specific information, such as an object's global location, must be relearned each time.
For each partitioned option o with sets of start and end states Io, S. S SxX, the agent re- re-partitions T, such that Pr(x' Izz o) Pr(æ') T., ox;,, 2t K. (1,2²) E 3g for C To- This forms partitioned subgoal options in both S and X. Denoting X C X' as the set of end states after re- re-pariitionig, the agent can ground the operator by appending to the precondition and À to the effect (if it differs from k), where x and À are treated as problem-specific propositions. Finally, these problem-specinee propositions must be linked with the grounded objects being acted upon. The agent therefore adds a precondition predicate conditioned on the identity of the grounded objects (see Appendix G for examples).
4 EXPERIMENTS
We first demonstrate our framework on the classic Blocks World domain (Section 4.1 While the high-level operators and predicates describing the domain are usually given, we show how such representation can be learned autonomously from scratch. We then demonstrate that our method scales to significantly harder problems by applying it to a high-ddmensiooal Minecraft task (Section 4.2). Finally, we investigate the transferability of the learned abstractions by transferring them to additional procedurally- generrated Minecraft tasks (Section 4.3). Owing to space constraints, we defer the exact implementation and domain details to the appendix.
4.1 LEARNING A REPRESENTATION OF BLOCKS WORLD
The Blocks World domain consists of number of blocks which can be stacked on top of one another by an agent (hand). The agent possesses options that allow it to pick up block (P i ck), put a block back on the table (Put), and stack one block on another (Stack). Blocks cannot be picked up if they are covered or if the hand is occupied, and can only be put down or stacked if already gripped. We consider the task consisting of three blocks A, B and C. where each block described by whether there is nothing, another block, or a table directly above or below it. This representation allows us to determine whether given block on a table, on another block, or grasped in the hand, and similarly whether another block has been stacked upon it. The hand is characterised by single boolean indicating whether it holding block. Thus a state is described by (fH: fA fg. f. corresponding to the hand and blocks' features respectively. Note that the agent is initially unaware that the blocks are functionally identical and can be treated interchangeably.
Using the approach outlined in Section 3.1 the agent partitions the options using transition data collected from the environment. This results in total of 15 partitions of the P Pick option, 3 partitions of the Put option, and 12 partitions of the Stack option It then fits classifier to each partition's initiation states, and density estimator to its terminating states. Finally, the agent generates a propositional PDDL using these learned preconditions and effects. Figure 2 illustrates a learned propositional operator, while the full PDDL, learned entirely from data, is provided in Appendix B
Generating a Propositional Model (Steps 1-2)
Generating a Lifted Typed Model (Steps 3-4) Using the effects from the propositional represen- tation, the agent determines that objects A, B and all possess the same effect profiles for all options and so can be grouped into a single type, while the hand belongs to its own type. The agent can now lift its representation by replacing the learned propositions with predicates parameterised by
See Appendix A for description of each partitioned option.
5

---

Under review as a conference paper at ICLR 2021
(:action Pick-partition-10 :paranetera :presondition (and (aymbol_10)
(symbol_15) (symboi_61)
:cffect (and (symbcl_ (aymbol_4)
(a) symbol 10
(b) symbol 1.1
(c) symboo..
(symbol
symbol_6)
(not symbol_101 (not symbo [51)
(g) Propositional PDDL operator for one of the P option partitions.
A
(d) symbol_3
(c) symbol_4
(f) symbol_1
Figure 2: The learned propositional operator for a Pick action describing picking off C. order to execute the action, the hand must be empty (symbol_1) _100), C must be on the table and covered by block (symbo1.15), and B must be on top of a block and uncovered (symbol.6), After execution, B is in the hand (symbol-3), C is on the table and clear (symbol-4) and the hand is full (symbol.1), We visualise the symbols by sampling from the propositional symbol, and randomly sampling the remaining independent state variables (since each symbol is distribution over subset of state variables). The transparency is due to the averaging over the independent state variables. Note that we must learn operator for every pair of blocks.
the above types. For example, after generating the model, there three propositions: AOnTable, BOnTable, l, and COnTable Since these are distributions over objects determined to be the same type, the agent replace them all with single predicate OnTable (X), which accepts block objects. As result, the agent can reduce the number of operators from 30 to 6, resulting in a more compact representation with smaller branching factor. Figure iillssrates how the propositional operator in Figure has been lifted to describe picking any block X off any block Y.
The number of learned operators is a function of the number of blocks in the domain. Since the propositional approach treats each block as its own unique object, it must learn the dynamics and interaction of each new block encounters. For blocks, this requires O(n2) operators. However, once the agent has learned the object types and constructed predicates based on these, it needs at most operators to represent the dynamics for any number blocks. See Appendix for the full parameterised PDDL.
W
(tact Pick-partitionss
:pecametera 1?w Lype0 ?x Lypel
2y typei)
:precondition (and (symbol_4 ?w)
(symbol _5 7x1 [aymbol_2 7y)1
(a) symbol.4
symbol_5
(c) symbol.2
reffect (symbol_) 7y1
(symbol_3 3 7x) (aymbol_ ?w)
W
(not (symbo 7y1 (not (symbol_4 7w1)
(not (symbol_5 2x)
(d) symbol.0
(e) symboi.3
(I) symbol.1
(g) Lifted PDDL operator Pick action.
Figure 3: The learned lifted operator for Pick action describing picking block off another. In order to pick up block Y, it must be on block X which itself is on the table, and the hand must be empty. As result, the hand is not empty, Y is now in the hand, and X is on the table and clear. type0 refers to the "hand" type, while type1 refers to the "block" type.
4.2 LEARNING A REPRESENTATION OF MINECRAFT TASK
In the above example, objects represented using pre-specified features and were sufficient to describe the environment dynamics. However, our approach is capable scaling beyond this simple case and learning these features from pixels. We demonstrate this in complex Minecraft task

---

Under review as a conference paper at ICLR 2021
(Johnson al. 2016) consisting of five rooms with various items positioned throughout. Rooms are connected with either regular doors which can be opened by direct interaction, or puzzle doors which require the agent to pull lever to open. The world is described by the state of each of the objects (given directly by each object's 's appearance as 600 x 800 RGB image), the agent's view, and current inventory. To simplify learning, we compress the state space by downscaling images and applying PCA (Pearson 1901] to a greyscaled version, preserving the top 40 principal components,
The agent is given high-level skills, such as ToggleDoor and WalkToItem. Execution is stochastic-opening doors occasionally fails, and the navigation skills are noisy in their execu- tion. To solve the task, an agent must first collect the pickaxe, use break the gold and redstone blocks and collect the resulting items. It must then navigate to the crafting table, where it uses the collected items to first craft gold ingots and subsequently clock. Finally, it must navigate to the chest and open it to complete the task. This requires a long-horizon, hierarchical plan- -te shortest plan that solves the task consists of 28 options consisting of hundreds of low-level continuous actions.
As previously, the agent begins by learning a model using the method outlined in Section 3.1 and in prior work (Konidaris et al. 2018 Ames et al. 2018). The agent partitions options using DBSCAN (Ester et al., 1996) to cluster option data based on terminating states. For each partitioned option, it then fits an SVM (Cortes & Vapnik] 1995) with Platt scaling (Plati 1999) to estimate the preconditions, and kernel density estimator Rosenblatt 1956) for effects, which are then used to construct the propositional PPDDL.
Generating a Propositional Model (Steps 1-2)
Generating a Lifted, Typed Model (Steps 3-4) Using the effects from the propositional represen- tation, the agent next groups objects into types based on their effect profiles. This is made easier because certain objects do not undergo effects under certain options. For example, the chest cannot be toggled, while door can, and thus it is immediately clear that they are not of the same type. Having determined the types, the agent replaces all similar propositions (where similarity is measured using the KL-divergence) with a single predicate parameterised by an object of that type.
Speccie Instantiation (Step 5) The agent now has a representation whose operators can be transferred between tasks. However, unlike Blocks World, complication arises because the object-centric state space is not Markov. For example, a state where all the doors are closed and the agent is in front of the first door is indistinguishable from state where the agent is in front of the second door. As described in Section (.33 the agent must ground the representations in the current task by incorporating additional problem-specific state variables to preserve the Markov property. These state variables are fixed across the family of MDPs; in this case, they are the agent's ryyz-lcation.
For each partitioned option, the agent again uses DBSCAN to cluster end states X' to form partitioned subgoal options in both S and X. Each of these clusters in X' is problem-specific proposition, which can be added to the learned operators to ground the problem. In Figure 4 we illustrate a learned operator for opening particular door, where the problec--pecific symbol has been tied to the door being opened in this manner. Without modifying the operator's parameter, would be possible to open any door at that location. The final plan discovered by the agent is illustrated by Figure 5
4.3 INTER-TASK TRANSFER IN MINECRAFT
We next investigate transferring operators between five grocedurally-generaae tasks, where each task differs in the location of the objects and doors; the agent cannot thus simply use plan found in one task to solve another. For given task, the agent transfers all operators learned from previous tasks, and continues to collect samples using uniform random exploration until produces a model which predicts that the optimal plan can be executed. Figure 6a shows the operators transferred between tasks, while Figure shows the number of interactions required to learn a model in a new task.
The minimum number of samples required to learn a model for a new task is bounded by the exploration strategy, since we must discover all problem-specine symbols to complete the model. Figure [b) shows that the number of samples required to leam model decreases over time towards this lower bound. Inter-task transfer could be further improved by leveraging the agent's 's existing knowledge perform non-uniform exploration, but we leave this to future work.
7

---

Under review as a conference paper at ICLR 2021
(saction toggle-bar-parttiion--
sperameters 12w type0 typel) sprecondition (and (notfailed)
(symbol FM) (symbol ex]
[- (1d 7x) (psymbol_241)
reffect [and (eymbol_6g fx)
(symbol_63 2w) (not (syabol_9 ?x)
(a) symbo1.37
(b) symbol_9 (c) paymbo1.24
(not (symbol_37 7w)1)
(I) A learned typed PDDL operator for partition of the Toggle-Doer option. The predicates underlined in red must be re- learned for each new task, while the rest of the operator can be safely transferred.
(d) symbol.64 (c) symbol_65
Figure 4: Our approach learns that, in order to open particular door, the agent must be standing in front of closed door (symbol_37) be closed (symbol_9) The effect of the skill is that the agent finds itself in front of an open door (s (symbol-64) and the door is open (symbol.65). type0 and typel refer the "agent" and "door" classes, while id is a fluent specifying the identity of the grounded door object, and is linked the problem-specifie symbol underlined in red.
partieular location (psymbol. 4), and the door must
ThvoggDoor
Walk
Albokk
WalkNork
TogpieDor
OpenCCest
WalkSeuth
Pickup
Codi
Figure Path traced by the agent executing different options while solving the first task.
60
Number of Opilon Executiors
950
Operators and instantistion
Number Operaaors
only
800
850
No Transfer
20
500
10
350
200
4
5
Number Tasks Seen
Number Tasks Seen
(a) Orange bars represent the number of operators that must be learned produce sufticiently rate model the number of operators transferred between tasks. As the number of tasks increases, the number of new operators that must be learned decreases.
(b) Number of samples required to learn suffi- ciently accurate models function of the number tasks encountered. The line represents the number samples required to learn all the oper- ators and the instantiation, while the green line accounts for the instantiation phase only.
the Blue bars represent
Figure 6: Results of learning and transferring high-level abstractions between tasks. We report the mean and standard deviation averaged over 80 runs with random task orderings.
RELATED WORK AND CONCLUSION
There has been work autonomously learning parameterised, transferable representations of skills. Ugur & Piater (2015) learn object-centric PDDL representations for robotic object manipulation tasks. They are able to learn object types which are similar to ours directly from data, but the object features specified prior learning, and discrete relations between object properties such as
8

---

Under review as a conference paper at ICLR 2021
width and height are given. Similarly, certain predicates manually inserted to generate sound representation, Asai (2019) learns object-centris abstractions directly from pixels, but it is unclear how to extend the approach to the stochastic setting. Furthermore, the representations lack soundness guarantees, and cannot be transformed into a language that can be used by existing task-level planners. Finney et al. (2002), Pasula et al. (2004) and Zettlemoyer et al. (2005) are able to learn operators that transfer across tasks, but the high-level symbols that constitute the state space are given. [James et al. (2020) learn PPDDL representation for planning using agent-relative and problem-specife data, but the operators are not lifted and there is no notion of objects or types, which limit generalisability.
Alternatively, object-oriented MDPs (Diuk et al. 2008 specify the state space as set of objects belonging to classes with associated attributes. The agent must learn the transition dynamics, which are usually restricted to small number of effects. We show how to learn an object-centric representation along with the object types, as well as the abstract high-level dynamics model.
By contrast, we have shown how to learn the vocabulary and the type system and the model from raw sensory input. Our representations are useful for planning, generalise across objects and can be transferred to new tasks. Although we have injected structure by assuming the existence of objects, this reflects the nature of many environments: fields such as computer vision assume that the world consists of objects, while there is evidence to suggest that infants do the same (Spelke) 1990). This assumption allows us to convert complex, high-dimensional environments to abstract representations that serve as input to task-level planners. Our approach provides an avenue for solving sparse-reward, long-term planning problems-suu such the MineRL compettoin (Guss et al. 2019 currently beyond the reach of model-free approaches.
REFERENCES
B. Ames, A. Thackston, and G.D. Konidaris. Learning symbolic representations for planning with parameterized skills. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018.
G. Andersen and G.D. Konidaris. Active exploration for learning symbolic representations. In Advances in Neural Information Processing Systems, pp. 5016-5026, 2017.
Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 583-591, 2019.
C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995.
C. Diuk, A. Cohen, and M.L. Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference on Machine Learning, pp. 240-247, 2008.
M. Ester, H. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In 2nd International Conference on Knowledge Discovery and Data Mining, volume 96, pp. 226-231, 1996.
R.E. Fikes and N.J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 213-4):189220 208, 1971.
S. Finney, N.H. Gardiol, L.P. Kaelbling, and T. Oates. The thing that we tried didn't work very well: deictic representation in reinforcement learning. In Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 154-161, 2002.
C. Guestrin, D. Koller, C. Gearhart, and N. Kanodia. Generalizing plans to new environments in relational MDPs. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, pp. 1003-1010. Morgan Kaufmann Publishers Inc., 2003.
W. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. Liebana, R. Salakhutdinov, N. Topin, et al. The MineRL competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv: 1904. 10079, 2019.

---

Under review as a conference paper at ICLR 2021
Mark K Ho, David Abel, Thomas L Griffiths, and Michael L Littman. The value of abstraction. Current Opinion in Behavioral Sciences, 29:111-116, 2019.
S. James, B. Rosman, and G.D. Konidaris. Learning to plan with portable symbols. In International Conference on Machine Learning, 2020.
M. Johnson, Hofmann, T. Hutton, and D. Bignell. The Malmo platform for artificial intelli- gence experimentation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pp. 1246-4247) 2016.
G.D. Konidaris, Scheidwasser, and A.G. Barto, Transfer in reinforcement learning via shared features. Journal of Machine Learning Research, 130Ma3):1333-1371, 2012.
G.D. Konidaris, L.P. Kaelbling, and T. Lozano-Pérez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61 (January):155-29,, 2018.
O. Marom and B. Rosman. Zero-shot transfer with deictic object-oriented representation in re- inforcement learning. In Advances in Neural Information Processing Systems, pp. 2297-2305, 2018.
D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL the planning domain definition language, 1998.
H. Pasula, L.S. Zettlemoer, and L.P. Kaelbling. Learning probabilistic relational planning rules. In Proceedings of the Fourteenth International Conference on Automated Planning and Scheduling, pp. 73 81, 2004.
K. Pearson. On lines and planes of closest fit to systems points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):539-57,, 1901.
J. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in Large Margin Classifiers, 10()):61-7,, 1999.
D. Precup. Temporal abstraction in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 2000.
N. Rosenblatt. Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statisticy, pp. 832-837, 1956.
E. Spelke. Principles of object perception. Cognitive Science, 14(1):29 -6, 1990.
R.S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 11((1-2):181-111 1999.
E. Ugur and J. Piater. Bottom-up learning of object categories, action effects and logical rules: from continuous manipulative exploration to symbolic planning. Proceedings of the 2015 IEEE International Conference Robotics and Automation, pp. 2627-2633, 2015.
H.L.S. Younes and M.L. Littman. PPDDL 1.0: An extension to PDDL for expressing planning domains with probabilistic effects. Technical report, 2004.
L.S. Zettlemoyer, H. Pasula, and L.P. Kaelbling, Learning planning rules in noisy stochastic worlds. In Proceedings of the Twentieth National Conference Artificial Intelligence, pp. 911-918, 2005.
10

---

Under review as a conference paper at ICLR 2021
A ENUMERATING SUBGOAL OPTIONS FOR THE BLOCKS WORLD DOMAIN
Given the description of the Blocks World domain in the main text, we must partition the given options (P ik,, Put and Stack) so that they adhere the subgoal condition. When there three blocks in the environment, we see that there are 30 partitioned options, which are described in the table below
Table I:
Descriptions of the different option partitions. The description of start and end states includes only the relevant information.
Option
partitions
Description of start states
Description of end states
PickoffTable (X)
3
X is on the table, is clear, and the hand empty.
X is grasped in the hand.
PickoffSingleslock (x, Y)
X on block Y which is the table, x is clear, and the hand is empty.
X is grasped in the hand and is clear and on the table.
PickOffDoubleBlock (x, Y)
X is on block Y which is on another block, is clear, and the hand is empty.
X is grasped in the hand and is clear and on another block.
StackOnSingleellock (x, Y)
x is in the hand, and Y is clear and on the table.
X is on block Y which on the table, and the hand is empty. X is on block Y which is another block, and the hand is empty.
StackOnDoubleslocck (x, Y)
x is in the hand, and
Y is clear and on other block.
Put (X)
3
X is grasped in the hand.
X is on the table and the hand is empty.
B PROPOSITIONAL PDDL DESCRIPTION FOR THE BLOCKS WORLD TASK
Below is the automatically generated propositional PDDL description of the Blocks World do- main with blocks. In practice, the agent generates this description with arbitrary names for the propositions, but for readability purposes we have manually renamed them to match their semantics.
(define (donain BlocksWorld)
(srequirementa :strips)
(spredi cates
faileei
(8Ialland)
(fandFull))
(COnBlock)
(BlnHand)
(ConTable)
(AOnTable)
(30aBlock)
(A0nBlock)
(B0aTable)
(CIalland)
(#zndimpty)
(30nTable BCavered)
#ennablle_Ravvred)
(AOnBlock )
Soonlloc__ccoveree
[conTable_tCaven
(ooveeel
11

---

Under review as a conference paper at ICLR 2021
(taction Pick_0 tparametera(1 :precondition land Illandmmpyy (AOnTable) (not.fileed)
reffect (end (AinHand) (HandPull) (not AOnTable) (not Handkmpty) (not Alntable) (not HandEmptyl)
(saction Pick_1 :parametera() :precendition land HlandEmpty) (AOnBIeck) (conBlcck_Cooeereee (notfailed))
reffect (end (DenBlock) (Ainsand) (Handrell) (not aimBlock) (act Handempty) (not conklock_ pok_occoeeed) A0nālock) (not HandRmpTy) (not (CnBiock_(Covered))
(raction 2ick_2 sparemeters sprecendition land [#andkmpty) (conTable, [antabll (80nslock) (notfailed)) seffect (and (RToflnn)) (ConTable) (HandFul (not RCaBlook) HandRmpty) (not DOnBlock) (not Handdnpp.y) (not Contable__Covezed))
(saction #ick_3
sparansters gprecondition HHandEmpty) (AOnTable ACovered: 150n5lock) (notrailed) reffect (ead (BInlland) (AOnTable) (llandFull) (not BCaBlock) (not HandEmpty) (nct sdniable_Acovesn (nct sonslock) (not Handimpay) (not ACoeeble
(saction Pick_4
:garemeteis(1 sprecondition land [HandEmpty) (AOnBlock) (B0xBlock (Coveree (notfeiled)) reffect
(#CnBlock) (Ainsand) (sandrell) (nct AdnBlock) Handampty) (net BonBlock__Covezeee (not A0nBlock) (not HandEmpzy) (not s(nslock_
(saction
sparemeters
:precendition [HandEmpzy) [Konslock_ACovered) ACovered: 180nslock) (notfailled) reffect (end (Biniland) (A0nBlock) (HendFull) (nct s0nBlock) (nct HandEmpty) (not sdnslock_Acovent (not sonslock) (not HandEmpry)
(not AGnRl AcnRlook_ACom ook__cceeed))))
dsaction sick_6 sparameters sprecendition land [Handkmpty) (Aonsleck) Muurabble_Wovvrree (notfailed)) seffect (and (RCnTahle) (AInHond) (HandFul1) (not AGnBlock) (not. HondEmpty) (not 50nTable (rrable_BCovered] (not Contable_BCoverell
40nBlook) (not HandEmpty)
tsaction Pick_?
sparemeters
sprecondition land [HandRmpty) (ROnTable) (not led))
reffect (BInlland) (HandFull) BOnTable) (sot flandEmpy)) BOnTable) (net HandEmpty!)
(saction Fick_8 :paremeters(! :precondition land [Handfmppy) (COnTable) (not
(CInlland) (MandFulll COnTable) (not [landEnpyy) COnTable) (net HandEmptyl)
(raction Pick_9 sparemeters(! sorecondition land IlladEmmpyy) (AOnTable ACoveredt Aooveeel 1C0nBLock) (not(ailed)) reffect (and (CInlland) (AOnfable) (laadFull) (not cCaBlock) HandBmpty)
(net Suntable_Acoveredd (not conslock) (not Handamp=y) (not Acntable_ACovered))
(saction #1ck_10
:paremeters(1
sprecondition [HandEmpzy) (AOnB1ozk) Centable_CCoverede (notfailed))
12

---

Under review as a conference paper at ICLR 2021
reffect
(CCnrable) (Ainsand) (Handrull) (nct admblock) Handampty) (not contablo_CCnvent (not A0nālock) (not HandEmpty) (not ConTable_ (
(saction Bick_11 sparametera :precendition [Handkmpty)
ICOnaleck)) (notfailedd)
seffect tand (CTrRand) (MOn8lock) (HandFy) (hot ConRlock) (not HandRmpty) (not ACuBlock (not COnBlock) (not Handdmp.yy (nct #CnBlock_
(saction Fick_12 aporametera :precondition land [HandEmptyl (conBlock) [BonTable_Boovrred) (notfailed)) reffect (end (BOnTable) (CIalland) (MladFa11)) (not CoaBlock) (not Handdmmty) (nct sontable__Coo BCovered) (not conslock) (not Handlmpty) (not scntablc__Covereet)
(saction 15ck_13 sparametera() :precondition land lllandEmpty) [COnBlock_CCover 120nBleck) (notfailed)) reffect (end (HinHand) (Conslock) (aandrell) (nct simBlock) (nct Handampty) (not conklock (Cavered) (not 80nslock) (not HandEmpoy) (not conBLoox__Covered)
Sonklock_CCaveredd
(saction Pick_14 sparemeters :precondition land (Handkmptyl (conslock) (80mBlock (BUoelaack_Boovardd)) (notfailed) seffect (and (RDnAlock) ICInHand) (HandFull) (not conRlock) (not HandRmpty) (not BoaBlock_DCovercee (nct J0nBlock_BCovered)
COnBlock) (not HandDmplyy
tsaction Put_15 sparsmeters
sprecondition land (flanFulll) (AInfland) (notrailed))
(and (ACnIable) IllandEmpty] fnot AInlland) (not MandFull) AInlland) (nct sandrull))
(saction Put_16 sparametera
sprecondition land illandFull) (BIalland) (not(ailed))
(and (BCniable) [HandEmpty] HandFull: lnss BInlland HandFulll (net sinsand))
(saction Fut_1? sparametera sprecondition land illandFull) (CIalland) (notfailed)) reffect (and (contable) (HandEmpty] Handrulll lnss CinHand) Handrull) (net cInsand))
(saction Stack_18 :gacenetesa() sprecondition land IllandFull) (CInlland) (BCnTablet las:failed) reffect (end (#Chtable_atevezer)) (conelock) (sandkmptyl Inoz Handrull) (net sonTable) (not cIneand). (not RandPull) (not BOnTable) (not CInRand))
(saction SLack_19
sparemeters :precendition 18andFull) (comBlock) (SInsendl Inosfailed)) seffect (and (RCnBlock) iconBl sec___cveeeeeddeeeeeoeeeeaeeeeee (HandRmptyl Inos RandFull) (not CCaBlock) (not Bialland) (not NaadPPlll (not (0nBleek) (net BInliand))
(saction Stack_20 sparametera
sprecondition land (HandFull) (A0mBlock) (BTnlandt Inostalled)) reffect (end (BCnBlock) (AOnBLeck ACovered) (landEmpty! taos BandPulll
(nct sinHand) (not 40nslock) (not sandrall) (not sinsand) (not 3CnBlock))
13

---

Under review as a conference paper at ICLR 2021
(taction Stack_21 tparametera(1 :precondition land IllandPull) (AInlland) (BOnTablet. ins:Selled)
reffect (end (Boniable_ateveen) (A0nBlock) (sendkmptyl lnoz AinHand) (not #andFull) sintable) (not AInHand) (not HandFull) (not SonTable)]
(saction Stack_22 :paremeters :precendition land (Handrull) (CinHand) (#Cnelocki Ino:feiled)
seffect (and BaaBllcc__coveeee (Con8lock) (HandRmpty) Inoz HandFull) (not B0nBlock) (not CInßand) (not HandFull) (not BOnBloox) (not CInHond))
(saction Stack_23 sporemsters sprecondition (HandFul (ConTablet (RTnlandt inosfafled)) reffect (and (BOnDlock) iCOnTable__Covee CCovered) (HandBmpLy) Ino. BandPull) (nct BInHand) conTable) HandPull) (not BInäand: (not COntable)
(taction stack_24 tparametera :precendition land IllandFull) (AInlland) (COnBlock! las:Sailed)
reffect [ccnslock_Cteveeee (A0nelock) (sandkmpty) Insz AinHand)
(not NandPull) (not COnBlock) (not AInlland) (not Handdulll) (nct CCmBlock)
Isaction SLack_25
sparemeters :precendition 18andrull) (ADstable) [Cinsendi Inssfailed)) reffect (end (COnBlock) (RonTable_Aooverdd) (HandEmpty) Inos BandFull) (not adntablee) (act Cinkand) (not HandPull) (not AdnTable) (not CInHand))
(saction $tack_26 aparamaters
sprecondition land (HandPull) (Ainlladd (contable: inssSeiled)
seffect (and Uuentable_Cteveeee (A0nBlock) (Handkmptyl Inot AInHand)
(not #andFull) ConTaa e) (not ATrHand) (not HandFul (not ConTable)
(saction Stack_27 sparemeters
:precondition land (HandFull) (A0mBlock) [CInSondi Inosfailed)) seffect (and (CDnBlock) [AOnBLook_]] totBlooo_ACoeeed] (HandEmptyl BandPull)
(not 3CaBlock) (not CIalland) (not flandPull) (not AOnBleek)
CinHand))
(saction Stack_28 sparameters gprecondition (HlndFull) (ACnTablel (BIolland: (no:talled)) reffect (end (BOnBlock) (AOnTable ACooeered (HandEmpty) taoz HandFulll (nct sinHand) (aot AOniable) sandrull) (not sinsand) (net scnTable)]
(saction Stack_29
:garemeteis sprecondition land (HandPull) (AinHland) (#CnBlock! Ins:failed))
(end aechBlock_stevese (Aonelock) (HandEmpty) Inot AInHand)
(net #andPul (not R0nRlook) (not ATrHand) (not HandFul fnot 3CaBlock)
14

---

Under review as a conference paper at ICLR 2021
C LIFTED PDDL DESCRIPTION FOR THE BLOCKS WORLD TASK
In contrast, the lifted representation learned below is far more compact. Here, operators are parame- terised by objects, which allows for better generalisation across instances with varying numbers of blocks. Figure shows how the number of partitioned options (and hence number of preconditions and effects) scales with the number of objects domain.
250
Propositional Lifted
200
Number operators
150
100
50
0
1
2
4
5
6
7
Number blocks
Figure The number of learned action operators as function of the number of blocks in the domain. Since the propositional approach treats each block as its own, unique object, it must learn the dynamies and interaction of each new block it encounters. For n blocks, this requires (n²) operators. However, if we learn the object types and construct predicates based on these, then need most operators to represent the dynamics for any number blocks.
Below we provide the learned representation for the domain. Again, we manually rename the predicates and types help with readability.
(domain BlocksWor
(:cequicemeats :strips :typing)
(stypes block] tspredicates
(BlockTnHand 7M block)
(HandFull 2w hand)
(Blockon8lock Pu blook)
(BlockOnTable block)
(HendEmpty ?M hand)
Blockcoveredd 28 block!
(Blcckon8llcck_ httpaockkkkoooooeeeeeooooo Zi block! (notfailed)
(saction Pick-partition-] block)
spacametess (?v sprecondition land inozfailed) (HandEmpty 2wt (BlockOnTable 2x))
seffect (end (BlockInRand 2x) (Kandrull 7M) (not (sloetonTable 2x)
(not (RandFmpty ?w)
(saction Pick perttitionn spareneters (?w :precondition land Inotfailed) (dandampty ?w1 rslockonslock ix)
2y
[BlackonBlock alockCavered ?y)
reffect (and (BlockCnBlock 7y) (BlockInßand 7x) (BangFull 74)
(not (BlockCaBlock 2x)) (not (llandEmpty 2w))
(net Bllcccnstocck_llocccoeee ?y))1
(saction #ick-partiticn-10
sparanetera (7M 7x ook ?y block!
15

---

Under review as a conference paper at ICLR 2021
sprecendition
land Inosfailed) (Handimpty [ (x)
jBlockOnBlack Py))
reffect (and (BlockInHand Py) (BlockonTable 7x) [flanFulll ?w)
(net (BlockOnBlock ?y1 (not (lladEeptty 2w))
(nct bllockontabie_Blocccoo 2x))1
(raction Put-partition-0
sparsmeters (?M hand 7x block)
:precondition land (notfailed) (BannPul:: ?w) (BlockIafland Px))
reffect (end (BlockOnTable 2x) (HandEmpty ?wl last IBlockInllend 2x))
(nct (Handsull (w))]
traction stack-paatittion-0
tparametera (?M hand 7x block Py block!
:precondition land lnotfailed) [llandPul: ?w) (BlockIalland ?x) (BlockOaTable 2y) reffect (end (not (Handrull 7w)) (not (BlockinTable ?yt) (not (RlockTnHand (x)1)
(Slcckonslock 7x) (Handkmpty 2w)
(saction Lon SLack partiinnt
spacemeters (?> hend 2x bleck 2y blockl sprecendition land inosfailed) (sandrul2 ?w)) (#lackinälock 7x) (BlockinHand 2y) reffect (and (BlocktnBlock ?y) (BlockOnBlock (not (MandFull (not (BlockIalland Py)1)
7x). (HandRmpty 7M)
A task might then be specified as follows:
(define (problem stack) (zdomain BlockeWorld)
(sobjects hand Hand Block
(iinit (BlockOnTable
(Blockinrable H)
(BlockO:Table C)
(Handampty hand) (notfailedd
(zgual (and (BlockOaBlock A)
(BlockoaBlock. htwp:oossllcc..lcccoeeeeeeeeeeee BlockCovered
Blockcovered 81))
16

---

Under review as a conference paper at ICLR 2021
D MINECRAFT TASK DETAILS
Our Minecraft tasks are procedurally generated, consisting of five rooms with various items positioned throughout. Rooms are connected with either regular doors which can be opened by direct interaction, puzzle doors which require the agent to pull lever to open. The world is described by the state of each of the objects (given directly by each object's appearance as a 600 x 800 RGB image), the agent's view, and current inventory. Figure illustrates the state of each object in the world the beginning one of the tasks.
Figure 8: The state of each object in the world the start of the task. From left right, the images represent the agent's 's point of view, the four doors, the pickaxe, the chest, and the redstone and gold blocks. The inventory not shown here.
The agent is provided with the following high-level skills:
(i) MalkToItem-thee agent will approach an item if is in the same room.
(ii) AttackBlock-the the agent will break block, provided it is near the block and holding the pickaxe.
(iii) PickupItem-the agent will collect the item it is standing in front of it.
(iv) WalkToNorthDoor the agent will approach the northern door in the current room. WalkToSouthDoor- the agent will approach the southern door in the current room. (vi) Wa halkKThooghDoor--rr the agent will walk through a door to the next room, provided the door is open.
vvii) CraftItem agent will create a new item from ingredients its inventory, provided it is near the crafting table.
(viii) OpenChest the agent will open the chest, provided it is standing in front of and possesses the clock.
(ix) Toggl eDoor agent will open close the door directly in front of it.
Execution is stochastic- opening doors occasionally fails, and the navigation skills are noisy in their execution.
E LEARNING A PORTABLE REPRESENTATION FOR MINECRAFT
In this section, we describe the exact details for learning a representation Minecraft task. Pseudocode for the approach (independent of the domain) is provided in Section
In order to learn a high-level representation, we first apply series preprocessing steps to reduce the dimensionality of the state space. We downscale images to 160 x 120 and then convert the resulting images to greyscale. We apply principal component analysis (Pearson) 1901) to batch of images collected from the different tasks and keep the top 40 principal components. This allows to represent each object (except the inventory, which is a one-hot encoded vector of length 5) as a vector of length 40.
17

---

Under review as a conference paper at ICLR 2021
Partitioning We collect data from task by executing options uniformly random. We record state transition data as well as, for each state. which options could be executed. We then partition options using the DBSCAN clustering algorithm (Ester et al. 1996) cluster the terminating states of each option into separate effects. This approximately preserves the subgoal property, as described in Section 2 and previous work (Andersen & Konidaris) 2017 Konidaris et al.] 2018] Ames et al.. 2018). For each pair of partitioned options, we check whether their is significant overlap in their initiating states (again using DBSCAN), If the initiating states overlap significantly, the partitions merged to account for probabilistic effects.
Preconditions Next, the agent learns a precondition classifier for each of these approximately partitioned options using an SVM (Cortes & Vapnik] 1995) with Platt scaling (Plati, 1999). We use states initially collected as negative examples, and data from the actual transitions as positive examples. We employ a simple feature selection procedure to determine which objects relevant to the option's precondition. We first compute the accuracy of the SVM applied to the object the option operates on, performing grid search find the best hyperparameters for the SVM using 3-fold cross validation. Then, for every other object in the environment, we compute the SVM's accuracy when that object's features added to the SVM. Any object that increases the SVM accuracy is kept. Pseudocode for this procedure is outline in Figure
Having determined the relevant objects, fit a probabilistic SVM to the relevant objects' data. Note that we learn a single SVM for a given precondition. Thus if the precondition includes two objects, then the SVM will learn a classifier over both objects' features jointly.
1: procedure FEATURESELECTION
2: Given: affected objects Mask, positive start states p. negative start states n, set of objects M
3: Fit classifier over only objects in the mask
4: classifier <- FiTCLAsSiFIER (start, negative, mask)
5: initScore +- classifier.scom
6: Keep +
7:
for each object E .M \ Mask do
8: classifier + FITCLASSIFIR (start, negative, Mask U {object))
newScore +- clossifier.score 10: if new:Score >> inilScore then
9:
11:
D Keep the object if improves the score Keep & Keep U {object]
12:
end if' end for
14:
return Mask U Keep
16:
end procedure
Figure 9: Pseudocode for simple feature selection procedure.
Effects A kernel density estimator (KDE) (Rosenblatt, 1956) with Gaussian kernel used to estimate the effect of each partitioned option. We learn distributions over only the objects affected by the option, learning one KDE for each object. We use a grid search with 3-fold cross validation find the best bandwidth hyperparameter for each estimator, We fit single KDE each object separately, since the state space has already been factored into these objects. Each of these KDEs is an abstract symbol in our propositional PDDL representation.
Propositional PDDL For each partitioned option, we now have classifier and set of effect distributions (propositions). However, to generate the PDDL, the precondition must be specified in terms of these propositions. We use the same approach as Konidaris al. (2018) to generate the PDDL: for all combinations of valid effect distributions, we test whether data sampled from their conjunction is evaluated positively by our classifiers. If they are, then that combination of distributions serves as the precondition of the high-level operator. This procedure is described in Figure TO
18

---

Under review as a conference paper at ICLR 2021
1: procedure BUILDPPDDLOPERATOR
2: Given: precondition classifier classifier, current effect effect, all effects Bffects
3: Operators +-
4: Symbols t- 0
5:
for each condidate E (Efffects) do samples t SAMPLE( condidate)
For all possible effect combinations Sample from the distributions Query the classifier with the data
6:
7:
prob PREDICT(classifier sample) if prob > 0 then
8:
9:
prob then
10:
& Construct the new operator with the existing effects operator t- {candidate, effect)
12:
else
13:
b Add a probabilistic failure case
newEffect fail, 1, with probability (1 - pprob)
14:
effect, with probability prob
15:
operalor + candidale, newBffecl)
end if
17:
Operators + Operators U {operator]
Symbols < Symbols U (candidate) U {effect)
19:
end if
20:
end for
21:
return Operators, Symbols
22:
end procedure
Figure 10: Pseudocode for constructing propositional PPDDL operators.
Type Inference To determine the type of each object, first assume that they all belong their own type. For each object, we compute its effect profile by extracting the effect propositions that occur under each option. Figure II illustrates this process.
For each pair of objects, we then determine whether the effect profiles are similar. This task is made easier because certain objects do not undergo effects with certain options. For example, the gold block cannot be toggled, while door can. Thus it is easy to see that they are not of the same type. To determine whether two distributions are similar, we simply check whether the Kl.-divergence less than certain threshold. Having determined the types, we can simply replace all similar propositions with predicate parameterised by an object that type, as described by Figure
1: procedure COMPUTREFFECTS
2: Given: object i, option o, PPDDL operators Operators
3: Get only the operators that model option o
4: Operators t- (operator Voperator Operators, REFFRSSTO(Operatie o)}
5: Effects (
6:
for each {:, effect) E Operators do
7:
Extract the effect propositions that refer to distributions over object i
8: Operalor..ffecl <- {prop Yprop E effect, REFERSTO(prop, i)}
9: Effects +- Effects (Operatorfffect)
10: end for
11: return Efffects
12:
end procedure
Figure 11: Pseudocode for computing the effect distributions under option for given object.
19

---

Efffett F.ffects J (estimator) Under review as a conference paper at ICLR 2021
1: procedure MERGE
2: Given: objects M. type T, PPDDL operators Operators, propositions Propositions
3: Find the first object matching the type
4: archetype 0
5:
for each object E M do
6:
if (STYPE(objeck)) T) then archelype object break end
7:
8:
9:
10:
end for
Remove propositions with objects of type T that are not the archetype
12:
Removed < {prop Vprop E Propositions, IST T), -REFERSTO(prup archetype)}
13:
Keep operators that do not contain the removed propositions Operators + (op Vop E Operators, Removed op = 0}
14:
15:
return Operators, Propositions \ Removed
16:
end procedure
Figure 12: Pseudocode for lifting propositions to typed predicates.
Problem-Specific Instantiation Finally, we again DBSCAN partition our subgoal options, but this time using problem-specift state variables. Each of these clusters is then added to our representation
cobblm--peecife proposition. To ground the operators, we add the start and end clusters (problem-specifie propositions) to the precondition and effects of the PPDDL operator. We also record the grounded object that appears the parameter list of each operator, and add precondition predicate (fluent) ensure that only those particular objects this final step, the agent would, for example, believe it can open any door while standing front of a door at particular location. We have thus linked the particular door to particular location in the domain.
be modified. Without
F PSEUDOCODE
Below we present pseudocode describing our approach to building a typed, object-centric PPDDL representation for arbitrary domain. Some subroutines used in the pseudocode below are outlined in the previous section.
1: procedure
2: Given: T' state-option transitions {(s,,,,,,,,,,,,, Fy, 01, s/, S i S T), set of objects M
3: Partition options into subgoal options
4: SubgoalOptions t
5:
for each E do
6:
-- {8 (8, 00,,00,%, D) + {s/ (...0,,'
Set of initial states for option Set terminating states for option (s,,o),s, | P+ (o, K, K Vs K, (8,,0,8 8' D)) & Start and end states for a partition SubgoalOptions
7:
E D}
8: for all K C such that Pr(s' 0) Pr(s'
I, s' E 3 do
9:
10:
SubgoolOptions U (P)
11:
end for end for
13:
Estimate preconditions and effects Preconditions, Effeect + 0 for each (., start, end) E SubgonlOptions do mask f- ASSPPUTTMAAKK(Saratt end)
14:
16:
> List the objects that change state
17:
negative + S start
18:
features +- VEXTURRSELLETTINN(me.. start, negative)
19:
classifier + FITCLASSIFER start, negative, features)
20:
Preconditions t- Preconditions U cclassifeer)
21:
estimator + FITESTIMATOR mask, end) b Fit over only objects that change
22:
20

---

Under review as a conference paper at ICLR 2021
23:
end for
24:
Build propositional PPDDL
25:
Operators, Propositions t-
26:
for each precondation, effect E (Preconditions Effects) do
27:
op, symbols + BULDDPDDLOPERATOR (precondition, oprrrpoDiD,,,RETTtooototttt,,,,, Operators < Operators U {op}
effect, Effects)
28:
29:
Propositions + Propositions U symbols
end for
31:
Infer object types
32:
PffProfile <- 0
33:
for each object m
34:
for each E O do EffProfilemm, o) < COMPUTEEFFECTS(m, o, Operators)
35:
36:
end for end for
37:
38:
Types (K) (.)Profile(m;;))) FiPrrofilemm E O, m, my KMKCM
39:
& Generate typed PPDDL
40:
TypedOperaaors. Predicates <- for each type Types do D Replace propositions and operators over objects of same type with lifted versions ops, predicates < MERGE(M, type, Operators, Proposilions) TypeatOssators +- TypedOOperators U ops
42:
43:
44:
45:
Predicates + Predicates U predicates
46:
end for
47:
Instantiate typed PPDDL new task
48:
for each {o, start, end) E SubgoalOptions do
49:
Lx 4- {ar Vs start, s' end, (8, x., 0, s) ED Bx < {a' Vs E start, s' E -endd E Ix. (s. s', D}
50:
for all Lx such that Pr(æ' Ii, Pr(r' Zj 2,,0 < fx' Vs E start, s' E end, x E K. (8, 0, ( E Predicates +- Predicates U U mask +- COMPUTEMASK > Link ppecifii symbols in precondition and effect to the affected objects TypeadOperttors +- GROUND ThypedOperators X, mask)
Lx, x' 3x do
52:
& Add mroblem-specifie symbols
54:
end)
> Computes the affected objects
55:
56:
57:
end for end for
58:
59:
return TypedOperators, Predicates
60:
end procedure
21

---

Under review as a conference paper at ICLR 2021
G VISUALISING OPERATORS FOR MINECRAFT
Here we illustrate some learned operators for the Minecraft tasks. To see all predicates and operators, please see the following URL: http://siits..ggll...... cwt.vieww.cc/ew..eeeooooo... mine pddl
(saction Open-Chest-partitl..e
:pacametera
Lype0
Lype6
Py type9)
:precondition
(sotfailed)
(aymbol_13 7w) (aymbol _A 7x)
(symbol_55 Py] (psymbol_ 61)
(a)
(b) symbol.4
(c)
seffact (and (aymbol
(aymbol_S)
symbol.13
symbol_55
(not (symbol_4 7x))
(symbol 2w)
(g) A learned typed PDDL operator for the Open-Chest: skill. The predicate underlined in red indicates problem-specific symbol that must be relearned for each new task, while the rest the operator can be safely transferred.
(d)
(c)
(f
psymbol_8
symbol.58
symbol.59
Figure 13: Our approach learns that, order to open a chest, the agent must be standing front of chest (symbol_13) the chest must be closed (symbol_4), the inventory must contain clock (symbo1_59) and the agent must be standing a certain location (psymbol_8) The result is that the agent finds itself in front of an open chest (symbol 58) and the chest is open (s ysymbol 59). tppe0 refers to the "agent" type, type 6 the "chest" type and type the "inventory" type.
(raction Walk partition :perameters [2w type0;
sprecendition (and (notfcilad) (aymbol_46 ?w)
(psymbol_00)
reffect (aymbol_ 7M) (paymbol_1)
(b)
(not: (symbol 46 74) (not [psymbo) 311)
symbol.46
psymbol.0
(e) Typed PDDL operator
partition Po option. The predicate underlined in red indi- cates problem-specific symhol that must be relearned for each new task, while the rest of the operator can be safely transferred.
(c)
(d)
symbo1.11
psymboll1
Figure 14: Abstract operator that models the agent walking to the crafting table. order to do so, the agent must be standing the middle of room (symbol-46 particular location (psymbol.0) As result, the agent finds itself in front of the crafting table (symbol.1) at a particular location (psymbol 1).
22

---

Under review as a conference paper at ICLR 2021
Isaction througgh-Door-pprrtttto---------
:parameters (2M type0 2x typel)
:presandition
(notfailed)
(symbo 1 _38 7M) (symbol 1464 ?x)
(- (id 2x) 1psymbol 241)
(a)
(b) symbol 64
reffect (and (symbol_50 2w)
(not (symbol 25r) [psymbot__22 12)
symbol
paymbol 24
(not (paymbo 2212))
(1) Typed PDDL operator for a partition the Throught Through-Deor option. The predicate under- lined in red indicates problem-speciee speciifc symbol that must be relearned for each new task, while the rest of operator can safely transferred.
(d)
(e)
symbol _50
psymbol 1ammmool_11
Figure 15: Abstract operator that models the agent walking through a door. In order to do so, the agent must be standing in front of an open door (symbol 1_38) particular location (psymbol. __24), and the door must be open (symbola 1_64). As a result, the agent finds itself in the middle of a room (symbol_50 at particular location (psymbol.12).
lraction Attack-partisicn----1- :parametera type0 2x type?)
:precordition land (nottailed]
(symbol_15 ?w) (symbol_2 Px)
(paymbol 1711 :effect (and (symbol_19 19 2x) (symbol_20 2w)
(a)
(b) symbol_2
(c)
(not (symbol 2x)
(not (symbol_15 2)))
symbo1.15
psymbol.1 _17
(f) Typed PDDL operator for partition of the Attack option. The predicate underlined red indicates problem-specific symbol that must be relearned for each new task, while the rest of the operator can be safely transferred.
(d)
symbo1 1
symbe1 20
Figure 16: Abstract operator that models the agent attacking an object. In order to do so, the agent must be standing in front of gold block (symbol 1.5) at particular location (psymbo 17), and the gold block must be whole (symbol_2). As result, the agent finds itself in front of disintegrated block (symbol_20) and the gold block is disintegrated (symbol_19)
23

---

Under review as a conference paper at ICLR 2021
H EXAMPLES OF FAILURE CASES
Below are some examples of errors that occur when constructing our abstract representation. Since there several phases involving clustering, classification and density estimation, expect various learning errors to occur throughout. These errors could have numerous causes, such as insufficient data or suboptimal hyperparameters.
H.1 PARTITIONING ERRORS
(a) Set of start states for one partition of the
(b) Set of end states for one partition of the Attaack option.
At Btackkk option.
(c) Set of start states for another partition of the At t ack option.
(d) Set of end states for another partition the At taak option.
Figure 17:
above example, the partitioning procedure has generated two partitioned options for breaking the gold block, where there should only be one. They are functionally equivalent, but because the strange shadows on the left the image patch and the subsequent PCA representation, the clustering algorithm has produced one extra partition.
(a) Set of states one partition of the ToggleDoor option.
(b) end one partition of the ToggleDoop option.
(c) Set of states for another partition of the loogl eloor option.
(d) Set of end for another partition of the Toggl elloor option.
Figure 18: In this example, the partitioning has clustered noisy samples into additional partition of the Togg eDoor option. While the top row shows the case where the state of the door changes from open closed, the bottom row is relatively useless noisy operator. We will subsequently learn a precondition and effect for this partition, but likely will not used by the planner.
24

---

Under review as a conference paper at ICLR 2021
H.2 PRECONDITIO//EATURE SELECTION ERRORS
(a) precondition attacking gold block. The top image represents the agent's 's view (in front of the block) while the bottom image is the state of the block (unbroken).
(b) The precondition for walking to a closed door. The top image represents the agent's 's view (in room) while the bottom image is the state of the door (closed) and the state of the inventory.
Figure 19: In the left example, the classifier predicts that the gold block can be broken when the agent in front of it. However, this is not quite correct, since the agent must also have the pickaxe to break the block. In this case, the issue occurs because the data only included states where the agent reached the gold block with the pickaxe. Therefore, the agent did not observe states where it was front of the block without the pickaxe, and thus concluded that the pickaxe is irrelevant to the precondition. In the right example, the classifier has overfitted to the data and predicts that the agent can only walk the door when it has the pickaxe.
H.3 PPDDL CONSTRUCTION ERROR
The quality of the PPDDL operators depends on how accurately the precondition classifiers and effect estimators are learned. Any error in learning result in imperfect PPDDL operators, seen below.
liaction Craft-partsttion-1-222
:parameters (7M typea 2x type
:presardition (and (notfailed)
(aymbo1_56 (symbol_29 7x)
[psymbol_111
reffect (probabilistic, 0.21
(a)
(b)
(c)
(not (notfailed)
symbol.56
symbol_29
psymbol.1
0.79
(and (symbol __ 7x) (aymbol_11 7w)
(not (symbol_43 _29 2x)
(not (symbol
(1) Typed PPDDL operator for a partition of the Cr a ft. option.
(d)
(e)
symbol.57
symbol_11
Figure 20: Abstract operator that models the agent crafting gold ingot. In order to do so, the agent must be standing front of the crafting table (symbol. 1_56) a particular location (psymbol 1_1), and must have the gold block in its inventory (symbol.29). As result, the agent finds itself in front of the crafting table (symbol.11), and now has a gold ingot in its inventory (symbol.51 This option is deterministic; however, due to estimation errors, the PPDDL operator predicts that will only succeed with probability 0.79.
25

---

Under review as a conference paper at ICLR 2021
H.4 TYPE INFERENCE ERROR
We observe that occasionally the procedure will not discover the correct types. In the example below, instead of discovering single type for all four doors, our approach predicts that one door is different from the others and is placed in its own class
Table 2: A grouping of objects into types. Note that one of the doors is allocated its own type.
Type
Name
Object(s)
0
Agent
0
Pickaxe
2
Doorl
2, 3, 4
3
Door2
5
4
Redstone Block
6
5
Gold Block
7
6
Chest
8
7
Inventory
9
26


Reviews:
[{"review_id": "yQcLA1uq1Nl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a planning approach that generalises across objects. Their approach groups objects into object types; if two objects have the same outcomes in planning then they are considered to be the same object. This is certainly a very interesting idea however I have some concerns about how this approach would work in more complex domains and how well the approach would generalise to imperfect object individuation.\n\nQuality:\n\nThe partitioning of initial states I_o  (for each option) is achieved by “clustering state transition samples based on terminating states”. Since the state is defined by a set {f_a, f_1, …., f_n} I have a few questions: (1) How do you ensure that the clustering is invariant to the order of the objects, f_i? (2) The clustering must depend heavily on how the state is represented?\n\nWhen learning the precondition model, if each object is treated independently, did the authors consider how this might be generalised to take into account interactions between objects? Similarly, when learning the effect model? \n\nIn this work the authors assume that the objects have already been individuated. While this is an important first step, it’s not clear how the approach would generalise if the object were imperfectly individuated. For example if only half of the door was visible in a particular frame? How would this approach scale to more complex observations?\n \nWere there any fail cases when partitioning options based on their terminal state? Particularly in the Minecraft setting?\n\nIt would be good to show examples of which object types were learned  and if there were any fail cases.\n\nFigure 6a is a very nice result — is it for BoxWorld of Minecraft? (Similar question for 6b).\n\n\nClarity:\nThe authors could make their contributions more clear in the introduction.\n\nFrom section 2 Is not clear what {f_a, f_1, …, f_n} are in the authors’ implementation.  The authors say that they are pixels, but go on to say that their “state space representation assumes that individual objects have already been factored into their constituent low-level attributes”. The first might suggest that f_i are image patches of objects while the latter suggests that they are vectors representing object attributes, it would help to clarify.\n\nThe authors use of the term representation is not well defined. From “a sound and complete abstract representation must necessarily be able to estimate the set of initiating and terminating states for each option” the authors’ definition of the term representation is not clear.\n\nAt the end of the “Partitioned Options” section, it would be good to clarify which approach will be used in this paper.\n\nFigure 1 is not very clear. \n\nThe feature selection process for determining which objects are are relevant to the precondition is only explained in the Appendix and may not be general. It is also not clear why this is needed? Why can the classifier not learn for itself which objects are useful and which to ignore? Is the implementation invariant to the ordering of the objects {f_i} for multiple i's?\n\nWhen learning the precondition model do you treat each object independently and predict Pre(o) for each object? It would be good to clarify this early on.\n\nIt would help if in the introduction and experiments sections the authors were more clear about what they want to measure to show that their approach is beneficial in task transfer.\n\n\n\nOriginality and Significance:\n\nThe authors’ novel contribution is to propose clustering objects based on their effect, allowing transfer between tasks. This is an interesting way to group objects and could enable future research towards using objects in RL, for example by relaxing assumptions that the objects have already been individuated. \n\n\nTypos:\n\nIn the introduction, PPDDL, is not defined.\n\n“initial and terminating states I_o and \\beta_o” -- \\beta_o is the probability of a terminating state, not the state itself.\n\nIn section 3.1 and Figure 1, steps 2: It’s not clear that you *generate* a positional forward model.\n\n“ to compute whether the skill can be executed there” —> do the authors mean option rather than skill here?\n\nFigure 4 is referenced before Figure 2.\n\nCheck the caption on Figure 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vlWDKaq0olE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- No Baseline Comparisons: Paper proposed a way to generate object level representation that can be used across the tasks with same objects. Authors claim that this should reduce the number of environment interactions required to solve the new task. However, there is no baseline comparison being done to figure out how sample efficient it is.\n\n- Paper assumed that tasks are solvable from set of provided options, will the method work of the task is out of provided options space ?\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vFpRbMvNbW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to learn a Probabilistic PDDL representation for tasks and generalize to new tasks with very small number of additional samples. It does that by first learning a compact and lifted representation across different training tasks that get the gist of multiple different objects, then converts it into symbolic representation (in PDDL) for future planning. \n\nI think the problem itself is super interesting and important: how to combine neural based and traditional approach remains a key challenge for AI community. And this paper seems to make a good attempt towards addressing it. \n\nHowever, I found this paper is hard to follow without any detailed descriptions of the actual algorithms.  The description is mainly high-level, many symbols/terms are mentioned but not defined, and there is no clue how to implement the idea. There seems to be 5 steps (Sec. 3.1-3.3 and Fig. 1). But it is not clear at all how the entire system work. Here I only list a few:\n\n1. what are the features defined in Sec. 4.1? \n2. Are the options o learned or pre-defined? \n3. How is the clustering /precondition classifier/feature selection in Sec 3.1 done? \n4. How are the profiles in Def. 1 computed? \n5. What is symbol_10 in Fig. 2? How is it learned? \n6. How does the algorithm replace three propositions AOnTable, BOnTable and COnTable with a single predicate InHand? This is the “lifting operation” mentioned in the paper but I didn’t find any algorithmic description. \n\nWhile the author defers “the exact implementation and domain details to the appendix”, I don’t find many details unfortunately, except for the learned/generated results.  The algorithm block in appendix F doesn't help much, with many undefined subroutines and symbols. \n\nGiven the difficulty in understanding the details, I feel that the paper is a bit premature for top-tier conference. \t\n\n======\n\nAfter reading the author's comments I still keep the score. \n\nAfter rebuttal/revision, the paper still has a lot of steps, many of them are human designed and are not well-defined. For example, in the author response, they said \"their effect distributions look similar, and so are merged into one class type\", what is the criterion to merge them together? In the revised paper, what is the procedure \"COMPUTEMASK\" defined?  In line 38 of the algorithm proposed in Appendix F in the revised paper, what does the sign of \"approximate equal\" mean?  Overall, this makes it hard to reproduce and it is not clear whether the proposed approaches can be applied to other problems than the specific tasks mentioned in the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3TFqNHh_oVm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach for object-centric representation learning for planning to accomplish complex tasks. The learned representations are at an abstract level, resulting in desirable knowledge transfer capabilities between tasks. The learned action knowledge is represented using PDDL. Experiments have been conducted using blocks world and Minecraft domains. Results show that the agent was able to learn useful operators (actions) and that learned actions can be applied to different tasks. \n\nThe object-centric idea is highlighted in the paper, though the work is more about learning symbols for abstraction, which is not new. The issue is that many of the \"learning abstraction\" works have been demonstrated in much more complex domains. For instance, the work of Konidaris et al 2018 (cited in the paper) has enabled robots to learn action preconditions and effects in the real world. In comparison, this work does not go beyond toy problems. \n\nThe developed approach is more like an integration of a few existing methods, and the connections among the pieces are rather weak (see Figure 1). For instance, once the actions are learned, it looks like the agent faces a planning problem, and the planner does not have a way to go back to improve its learned representation. Since each component introduce its own errors, there's the cumulative error that can potentially make the whole system rather unstable. Some co-learning functionalities will be good for future work. \n\nThe experiment section is relatively weak. For instance, the blocks world domain was mostly used as a demonstration platform. There were a couple of examples presented, while there were no statistical results discussed in the paper. Figure 6 is on the Minecraft domain, where the baseline (such as no transfer) is very weak. The results are not convincing to support the claims on transferability and learning efficiency. \n\n-------\n\nThe reviewer appreciates very much, and has read the response letter, which is mostly about clarity though it's not the main concern of the reviewer. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper proposes a general framework for learn object-centric abstractions represented using PPDDL (a probabilistic planning language).  The work assumes that objects and their attributes / features are identified.  The key contribution of the paper appears to be proposing to group individual objects into object types based on whether objects have the same outcome in planning. Using the learned object types, it would then be possible to transfer learned operators from one task to another.  The framework is demonstrated on block world (blocks are stacked on top of on another) and minecraft.  


Review Summary: Initially, the submission received negative to borderline reviews with R4 being the most positive (score 6), and R1, R2, R3 being more negative (scores 4, 3, 5).  After discussion, R4 lowered their score to 4 and indicated that they felt the work was not ready for acceptance at ICLR.  Overall, there was limited discussion by the reviewers.  Reviewers (R2,R4) found the direction of the work promising and interesting.  After the author response, reviewers indicated that the revision and author response clarified some points, but believe that the work is not yet ready for acceptance, as 1) there is a significant amount of hand-crafting required and 2) parts of the approach is still not clearly specified.  

Clarity: As some reviewers note, the description of the framework is at a very high level, making it difficult to follow with missing details on specific details of how the object types are groups.  The specific aspect of the work that is novel is also not clearly stated, thus making it difficult to judge the originality and significance of the work.  After revision (the authors added brief paraphrase to the introduction to clarify the contribution, and additions to the appendices providing more details on how the difference steps work for the Minecraft scenario as well as failure cases), the manuscript is improved but the overall manuscript is still difficult to follow.

Pros:
- Interesting and important problem (combining probabilistic/neural approaches with symbolic approaches) that is timely and deserves attention
- The idea of clustering objects based on their effect is interesting (R4).  
- The framework proposed by the paper is interesting and potentially useful direction and can stimulate followup work 

Cons:
- The paper is difficult to follow with symbols/terms that are not clearly defined and missing details. (R1) The specific contributions of the work, wrt prior work, is also not clearly stated.
- The novelty/contribution of the work on top of existing work (Konidaris et al 2018, Ugur & Piater 2015, etc) is not that clear (R2)
- The experimental setup is weak with limited comparisons and no statistical results. Overall, reviewers felt the results are not convincing enough to support claims on transferability and learning efficiency.
- Lack of baselines comparisons (R3).  In the rebuttal, the authors argue that there is no appropriate baselines.  
- The set of steps that is involved is fairly complex (R1), with many important details provided in the appendix
- There are concerns about the generalization of the approach as many of the steps are handcrafted (R1, R4).  In the provided scenarios, many of the steps, including the set of provided options, and representation of objects, are manually designed.  

Recommendation:
The AC agrees with the reviewers that the paper is not ready for acceptance to ICLR.  It is the AC's opinion that the work addressing a very interesting problem and 
would potentially be of interest to the community.  However, the exposition of the paper needs to be improved so that 1) the contribution of the work over prior work (Konidaris et al 2018, Ugur & Piater 2015, etc) is clearer 2) the assumptions and details of the proposed method is also clearer and easier to follow.  The authors are encouraged to improve their work and resubmit to an appropriate venue.  
Decision: Reject