Autonomous Learning of Object-Centric Abstractions for High-Level Planning

1 INTRODUCTION . Model-based methods are a promising approach to improving sample efficiency in reinforcement learning ( RL ) . However , they require the agent to either learn a highly detailed model—which is infeasible for sufficiently complex problems ( Ho et al. , 2019 ) —or to build a compact , high-level model that abstracts away unimportant details while retaining only the information required to plan . This raises the question of how best to build such an abstract model . While recent advances have shown how to learn models of complex environments , they lack theoretical guarantees and require millions of sample interactions ( Schrittwieser et al. , 2020 ; Hafner et al. , 2021 ) . Fortunately , recent work has shown how to learn an abstraction of a task that is provably suitable for planning with a given set of skills ( Konidaris et al. , 2018 ) . However , these representations are highly task-specific and must be relearned for any new task , or even any small change to an existing task . This makes them fatally impractical , especially for agents that must solve multiple complex tasks . We extend these methods by incorporating additional structure—namely , that the world consists of objects , and that similar objects are common amongst tasks . For example , when we play video games , we solve the game quickly by leveraging our existing knowledge of objects and their affordances ( such as doors and ladders which occur across multiple levels ) ( Dubey et al. , 2018 ) . Similarly , robot manipulation tasks often use the same robot and a similar set of physical objects in different configurations . This can substantially improve learning efficiency , because an object-centric model can be reused wherever that same object appears ( within the same task , or across different tasks ) and can also be generalised across objects that behave similarly—object types . We assume that the agent is able to individuate the objects in its environment , and propose a framework for building portable object-centric abstractions given only the data collected by executing high-level skills . These abstractions specify both the abstract object attributes that support high-level planning , and an object-relative lifted transition model that can be instantiated in a new task . This reduces the number of samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously seen object types . We make the following contributions : under the assumption that the agent can individuate objects in its environment , we develop a framework for building portable , object-centric abstractions , and for estimating object types , given only the data collected by executing high-level skills . We also show how to integrate problem-specific information to instantiate these representations in a new task . This reduces the samples required to learn a new task by allowing the agent to avoid relearning the dynamics of previously-seen objects . We demonstrate our approach on a Blocks World domain and a 2D crafting domain , and then apply it to a series of Minecraft tasks where an agent autonomously learns an abstract representation of a high-dimensional task from raw pixel input . In particular , we use the probabilistic planning domain definition language ( PPDDL ) ( Younes & Littman , 2004 ) to represent our learned abstraction , which allows for the use of existing task-level planners . Our results show that an agent can leverage these portable abstractions to learn a representation of new Minecraft tasks using a diminishing number of samples , allowing it to quickly construct plans composed of hundreds of low-level actions.1 2 BACKGROUND . We assume that tasks are modelled as semi-Markov decision processesM = 〈S , O , T , R〉 where ( i ) S is the state space ; ( ii ) O ( s ) is the set of temporally-extended actions known as options available at state s ; ( iii ) T describes the transition dynamics , specifying the probability of arriving in state s′ after option o is executed from s ; and ( iv ) R specifies the reward for reaching state s′ after executing option o in state s. An option o is defined by the tuple 〈Io , πo ; βo〉 , where Io is the initiation set specifying the states where the option can be executed , πo is the option policy which specifies the action to execute , and βo the probability of the option terminating in each state ( Sutton et al. , 1999 ) . We adopt the object-centric formulation from Ugur & Piater ( 2015 ) : in a task with n objects , the state is represented by the set { fa , f1 , f2 , . . . , fn } , where fa is a vector of the agent ’ s features and fi is a vector of features particular to object i . Note that the feature vector describing each object can itself be arbitrarily complex , such as an image or voxel grid—in one of our domains , we use pixels . Our state space representation assumes that individual objects have already been factored into their constituent low-level attributes . Practically , this means that the agent is aware that the world consists of objects , but is unaware of what the objects are , or whether multiple instantiations of the same object are present . It is also easy to see that different tasks will have differing numbers of objects with potentially arbitrary ordering ; any learned abstract representation should be agnostic to this . 2.1 STATE ABSTRACTIONS FOR PLANNING . We intend to learn an abstract representation suitable for planning . Prior work has shown that a soundand complete abstract representation must necessarily be able to estimate the set of initiating and terminating states for each option ( Konidaris et al. , 2018 ) . In classical planning , this corresponds to the precondition and effect of each high-level action operator ( McDermott et al. , 1998 ) . The precondition is defined as Pre ( o ) = Pr ( s ∈ Io ) , which is a probabilistic classifier that expresses the probability that option o can be executed at state s. Similarly , the effect or image represents the distribution of states an agent may find itself in after executing an option from states drawn from some starting distribution ( Konidaris et al. , 2018 ) . Since the precondition is a probabilistic classifier and the effect is a density estimator , they can be learned directly from option execution data . We can use preconditions and effects to evaluate the probability of a sequence of options—a plan— executing successfully . Given an initial state distribution , the precondition is used to evaluate the probability that the first option can execute , and the effects are used to determine the resulting state distribution . We can apply the same logic to the subsequent options to compute the probability of the entire plan executing successfully . It follows that these representations are sufficient for evaluating the probability of successfully executing any plan ( Konidaris et al. , 2018 ) . Partitioned Options For large or continuous state spaces , estimating Pr ( s′ | s , o ) is difficult because the worst case requires learning a distribution conditioned on every state . However , if we assume that terminating states are independent of starting states , we can make the simplification Pr ( s′ | s , o ) = Pr ( s′ | o ) . These subgoal options ( Precup , 2000 ) are not overly restrictive , since they refer to options that drive an agent to some set of states with high reliability . Nonetheless , many options are not subgoal . It is often possible , however , to partition an option ’ s initiation set into a finite number of subsets , so that it is approximately subgoal when executed from any of the individual subsets . That is , we partition an option o ’ s start states into finite regions C such that Pr ( s′ | s , o , c ) ≈ Pr ( s′ | o , c ) , c ∈ C ( Konidaris et al. , 2018 ) . 1More results and videos can be found at : https : //sites.google.com/view/mine-pddl Factors We adopt the frame assumption , which states that aspects of the world not explicitly affected by an agent ’ s action remain the same ( Pasula et al. , 2004 ) . Prior work leverages this to learn a factored or STRIPS-like ( Fikes & Nilsson , 1971 ) representation by computing the option ’ s mask : the state variables explicitly changed by the option ( Konidaris et al. , 2018 ) . In our formulation , the state space is already factorised into its constituent objects , so computing the mask amounts to determining which objects are affected by a given option . 3 LEARNING OBJECT-CENTRIC REPRESENTATIONS . Although prior work ( Konidaris et al. , 2018 ) allows an agent to autonomously learn an abstract representation supporting fast task-level planning , that representation lacks generalisability—since the symbols are distributions over states in the current task , they can not be reused in new ones . This approach can be fatally expensive in complex domains , where learning an abstract model may be as hard as solving a task from scratch , and is therefore pointless if we only want to solve a single task . However , an agent able to reuse aspects of its learned representation can amortise the cost of learning over many interactions , accelerating learning in later tasks . The key question is what forms of representation support transfer in this way . We now introduce an object-centric generalisation of a learned symbolic representation that admits transfer in tasks when the state space representation consists of features centred on objects in the environment . This is common in robotics , where each object is often isolated from the environment and represented as a point cloud or subsequently a voxelised occupancy grid . Our approach builds on a significant amount of machinery , involving clustering , feature selection , classification and density estimation . We summarise our proposed approach in Figure 1 and provide a high-level description in the remainder of this section , but provide pseudocode and specific practical details in the appendix . 3.1 GENERATING A PROPOSITIONAL MODEL ( STEPS 1–2 ) ( AS IN KONIDARIS ET AL. , 2018 ) . The agent begins by collecting transition data using an exploration policy . The first step is to partition the options into approximately subgoal options . For each option o and empirical sets of initial and terminating states Ĩo and β̃o , the agent partitions Ĩo into a number of disjoint subsets , such that for each subset K ⊆ Ĩo , we have Pr ( s′ | si , o ) = Pr ( s′ | sj , o ) ∀si , sj ∈ K , s′ ∈ β̃o . In other words , the effect distribution of the option is identical , independent of the state in K from which it was executed . In practice , this can be approximated by first clustering state transition samples based on terminating states , and then assigning each cluster to a partition . Finally , pairs of partitions whose initiating states overlap are merged to handle probabilistic effects ( Konidaris et al. , 2018 ) . The agent next learns a precondition classifier for each approximately partitioned option . For each partition , the initiating states are used as positive examples , and all other states are treated as negative ones . A feature selection procedure next determines which objects are relevant to the precondition , and a classifier is fit using only those objects . A density estimator is then used to estimate the effect distribution for each partitioned option . The agent learns distributions over only the objects affected by the option , learning one estimator per object . Together these effect distributions form our propositional PPDDL vocabulary V . To construct a PPDDL representation for each partitioned option , we must specify both the precondition and effects using the state distributions ( propositions ) in V . The effects are directly specified using these distributions , and so pose no problem . However , the estimated precondition is a classifier rather than a state distribution . The agent must therefore iterate through all possible effect distributions to compute whether the skill can be executed there . To do so , we denote P as some set of propositions in V , and G ( s ; P ) as the probability that a low-level state s is drawn from the conjunction of propositions in P . Then , for an option with learned classifier Îo , we can represent the precondition with every P ∈ ℘ ( V ) such that ∫ S Îo ( s ) G ( s ; P ) ds > 0 , where ℘ ( V ) denotes the powerset of V . In other words , the agent considers every combination of effect state distributions and draws samples from their conjunction . If these samples are classified as positive by Îo , then the conjunction of P is used to represent the precondition . The preconditions and effects are now specified using distributions over state variables , where each distribution is a proposition—this is our PPDDL representation , which is sound and complete for planning .

Summary: This paper proposes a method for learning an object-centric symbolic representation of an environment that allows for planning. It extends the framework introduced in Konidaris et al. (2018), which learns a symbolic representation of an environment that can be expressed using a PDDL for planning. Importantly, the method presented in Konidaris et al. (2018) does not make assumptions about structure in the environment and how different states may relate to one another. As a consequence, the learned representations are highly tasks-specific and transfer/generalization to new tasks and across states is not possible.   In this paper it is assumed that the world consists of objects and that similar objects are common across tasks. This enables object-centric representations (and the associated model) to be reused across tasks, and organizing objects according to 'object types' that behave similarly when acted upon. This is expected to improve learning efficiency and generalization/transfer to new tasks.   The method presented here incorporates techniques for merging objects into object types, and integrating problem-specific information to ground these representations into specific tasks. The working of these techniques is validated experimentally, and it is shown how greater sample efficiency can be obtained when generalizing to new tasks.