Generating Plannable Lifted Action Models for Visually Generated Logical Predicates

1 INTRODUCTION . Learning a high-level symbolic transition model of an environment from raw input ( e.g. , images ) is a major challenge in the integration of connectionism and symbolism . Doing so without manually defined symbols is particularly difficult as it requires solving both the Symbol Grounding ( Harnad , 1990 ; Taddeo & Floridi , 2005 ; Steels , 2008 ) and the Action Model Learning/Acquisition problem . Recently , seminal work by Asai & Fukunaga ( 2018 , Latplan ) that learns discrete planning models from images has opened the door to applying symbolic Classical Planning systems to a wide variety of raw , noisy data . Latplan uses discrete variational autoencoders to generate propositional latent states and its dynamics ( action model ) directly from images . Unlike existing work , which requires several machine learning pipelines ( SVM/decision trees ) and labeled inputs ( e.g. , a sequence of high-level options ) ( Konidaris et al. , 2014 ) , Latplan is an end-to-end unsupervised neural network that requires no manually labeled inputs . Numerous extensions and enhancements have been proposed : Causal InfoGAN ( Kurutach et al. , 2018 ) instead uses GAN framework to obtain propositional representations . Latplan ’ s representation was shown to be compatible with symbolic Goal Recognition ( Amado et al. , 2018 ) . First-Order State AutoEncoder ( Asai , 2019 , FOSAE ) extends Latplan to generate predicate symbols . Cube-Space AutoEncoder ( Asai & Muise , 2020 , CSAE ) regularized the latent space to a particular form which directly exports to a learned propositional PDDL model ( Fikes et al. , 1972 ) . Discrete Sequential Application of Words ( DSAW ) learns a plannable propositional word embedding from a natural language corpus ( Asai & Tang , 2020 ) . In this paper , we obtain a lifted action model expressed in First-Order Logic ( FOL ) , which is a superset of object-centric ( property-based ) representation that Machine Learning community recently began to pay attention to1 , but has long been the central focus of the broader AI community . In propositional action models , the environment representation is a fixed-sized binary array and does not transfer to a different or a dynamically changing environment with a varying number of objects . In contrast , lifted FOL representations are generalized over objects and environments , as we demonstrate in Blocksworld with different number of blocks , or Sokoban with different map sizes . We propose Lifted First-Order Space AutoEncoder ( FOSAE++ ) neuro-symbolic architecture , which learns a lifted PDDL action model by integrating and extending the FOSAE , the CSAE and the Neural Logic Machine ( Dong et al. , 2019 , NLM ) architectures . The overall task of our system is illustrated in Fig . 1 . The system takes a transition dataset containing a set of pairs of raw observations which are single time step away . Each observation consists of multiple visual segmentations of the objects . It learns a lifted action model of the environment 1e.g. , ICML workshop on Object-Oriented Learning https : //oolworkshop.github.io/ by generating the symbols and emits a PDDL ( Haslum et al. , 2019 ) encoding for state-of-the-art planning systems . Contribution Table 1 contains a taxonomy of existing model acquisition systems in chronological order . FOSAE++ is the first system that satisfies all features readily available in symbolic action model acquisition systems , while not relying on human-derived symbols . FOSAE++ generates unnamed symbols by itself — Effectively addressing the long-standing Knowledge Acquisition bottleneck ( Cullen & Bryman , 1988 ) and the Symbol Grounding problem , showing a future direction for high-level symbolic autonomy . 2 PRELIMINARIES AND BACKGROUND . We denote a multi-dimensional array ( tensor ) in bold and its elements with a subscript ( e.g. , x ∈ RN×M , x2 ∈ RM ) , an integer range n ≤ i ≤ m by n .. m , a concatenation of tensors a and b in the last axis by a ; b , and the i-th data point of a dataset by a superscript i which we may omit for clarity . We use the same symbol for a set and its size ( e.g. , S , and not |S| ) to avoid the clutter . Finally , B = [ 0 , 1 ] ⊂ R. We assume background knowledge of discrete VAEs with continuous relaxations ( included in the appendix Sec . A.1 ) , such as Gumbel-Softmax ( GS ) and Binary-Concrete ( BC ) ( Jang et al. , 2017 ; Maddison et al. , 2017 ) . Their activations are denoted as GS and BC , respectively . 2.1 LIFTED STRIPS/PDDL PLANNING . Planning Domain Description Language ( PDDL ) is a modeling language for a Lifted STRIPS planning formalism ( Fikes et al. , 1972 ) and its extensions ( Haslum et al. , 2019 ) . Let F ( T ) be 1 ( Yang et al. , 2007 ; Cresswell et al. , 2013 ; Aineto et al. , 2018 ; Zhuo et al. , 2019 ; Cresswell & Gregory , 2011 ; Mourão et al. , 2012 ; Zhuo & Kambhampati , 2013 ) 2Konidaris et al . ( 2014 ) requires sequences of high-level options to learn from , such as [ move , move , interact , · · · ] in Playroom domain . Causal InfoGAN can not deterministically enumerate all successors ( a requirement for search completeness ) due to the lack of action symbols and should sample the successors . James et al . ( 2020b ) ’ s PDDL output is limited to unary predicates / properties of objects , thus can not model the interactions between objects . Also , it requires sequences of high-level options such as [ WalkToItem , AttachBlock , WalkToItem , · · · ] in the Minecraft domain . a formula consisting of logical operations { ∧ , ¬ } and a set of terms T . For example , when T = { have ( I , food ) , full ( I ) } , then have ( I , food ) ∧ ¬full ( I ) ∈ F ( T ) . We denote a lifted STRIPS planning problem as a 5-tuple 〈O , P , A , I , G〉 . O is a set of objects ( 3 food ) , P is a set of predicates ( 3 full ( x ) ) , and A is a set of lifted actions ( 3 eat ) . Each predicate p ∈ P has an arity # p ≥ 0 . Predicates are instantiated/grounded into propositions P ( O ) = ⋃ p∈P ( { p } ×O × # p. . .×O ) , such as have ( I , food ) . A state s ⊆ P ( O ) represents truth assignments to the propositions , e.g. , s = { have ( I , food ) } represents have ( I , food ) = > . We can also represent it as a bitvector of size ∑ pO # p. Each lifted action a ( X ) ∈ A has an arity # a and parameters X = ( x1 , · · · , x # a ) , such as eat ( x1 , x2 ) . Lifted actions are instantiated into ground actions A ( O ) =⋃ a∈A ( { a } ×O × # a. . .×O ) , such as eat ( I , food ) . a ( X ) is a 3-tuple 〈PRE ( a ) , ADD ( a ) , DEL ( a ) 〉 , where PRE ( a ) , ADD ( a ) , DEL ( a ) ∈ F ( P ( X ) ) are preconditions , add-effects , and delete-effects : e.g. , eat ( x1 , x2 ) = 〈 { have ( x1 , x2 ) } , { full ( x1 ) } , { have ( x1 , x2 ) } 〉 . The semantics of these three elements are as follows : A ground action a† ∈ A ( O ) is applicable when a state s satisfies PRE ( a† ) , i.e. , PRE ( a† ) ⊆ s , and applying an action a† to s yields a new successor state a† ( s ) = ( s \ DEL ( a† ) ) ∪ ADD ( a† ) , e.g. , eat ( I , food ) = “ I can eat a food when I have one , and if I eat one I am full but the food is gone. ” Finally , I , G ⊆ P ( O ) are the initial state and a goal condition , respectively . The task of classical planning is to find a plan ( a†1 , · · · , a†n ) which satisfies a†n ◦ · · · ◦ a † 1 ( I ) ⊆ G . 2.2 NEURAL PROPOSITIONAL/ACTION SYMBOL GENERATION WITH LATPLAN . Latplan is a framework for domain-independent image-based classical planning ( Asai & Fukunaga , 2018 ) . It learns a propositional state representation and transition rules entirely from image-based observations of the environment with discrete VAEs and solves the problem using a classical planner . Latplan is trained on a transition input Tr : a set of pairs of raw data randomly sampled from the environment . The i-th transition ( oi,0 , oi,1 ) ∈ Tr is a pair of observations made before and after an unknown high-level action is performed . Once trained , Latplan can process a planning input ( oI , oG ) , a pair of raw images corresponding to an initial and goal state of the environment . The output of Latplan is a data sequence representing the plan execution ( oI , . . .oG ) that reaches oG from oI . While the original paper used an image-based implementation , conceptually any form of temporal data is viable for this methodology , e.g. , an NLP corpus ( Asai & Tang , 2020 ) . The latest Latplan Asai & Muise ( 2020 ) has a training phase and a planning phase . In the training phase , it trains an end-to-end neural network called Cube-Space AutoEncoder ( CSAE ) on Tr ( Fig . 2 , top left ) . CSAE is a variational autoencoder modeled by binary and categorical random variables each representing the propositional states and the actions in the classical planning . The dynamics modeled by these actions directly compiles into a PDDL model . The network combines BinaryConcrete VAE to produce binary state representation , and a Gumbel-Softmax VAE to produce a categorical bottleneck layer which assigns a categorical label to each input . Let o0 and o1 be a pair of observed states in a transition , z0 and z1 be the corresponding binary latent states , and a be the one-hot vector that represents a discrete action label assigned to the transition . CSAE is a variational autoencoder network that can be formalized as follows : ( encoder ) z0 , z1 = ENCODE ( o0 ) , ENCODE ( o1 ) ∈ BF ( action assignment/clustering ) a = ACTION ( z0 , z1 ) ∈ BA ( learning the dynamics ) z∼1 = APPLY ( z0 , a ) ∈ BF ( decoder ) o∼0 , o∼1 , o∼ 1 = DECODE ( z0 ) , DECODE ( z1 ) , DECODE ( z∼1 ) where ENCODE , DECODE , ACTION , APPLY are arbitrary multilayer perceptrons . The outputs of ENCODE and APPLY are activated by Binary Concerete , and the output of ACTION is activated by Gumbel-Softmax of A categories ( hyperparameter ) . Assuming a certain set of prior distributions , a lower bound ( ELBO ) of the log likelihood of observing a pair of states ( o0 , o1 ) can be as derived follows ( Appendix Sec . A.3.3 ) : log p ( o0 , o1 ) ≥ −DKL ( q ( a|o1 , z0 , z1 ) ||p ( a|z0 , z1 ) ) −DKL ( q ( z∼1|o1 , a , z0 , z1 ) ||q ( z1|o1 ) ) −DKL ( q ( z0|o0 ) ||p ( z0 ) ) −DKL ( q ( z1|o1 ) ||p ( z1 ) ) + log p ( o0|z0 ) + log p ( o1|z∼1 , a , z0 , z1 ) After the training , it generates a propositional classical planning problem ( zI , zG ) from a planning input ( oI , oG ) and export it into PDDL files together with the action model obtained in ( 2 ) , which are then solved by Fast Downward ( Helmert , 2006 ) , an optimized C++-based solver independent from the neural network . Finally , it obtains a step-by-step , human-comprehensible visualization of the plan execution by decoding the intermediate states of the plan into images . The A-dimensional one-hot categorical variable ai in the network performs a clustering on the state transitions with the maximum number of clusters A specified as a hyperparameter . The cluster ID is used as its action symbol . The clustering is performed by its encoder , ACTION ( zi,0 , zi,1 ) = ai , which takes a propositional state pair ( zi,0 , zi,1 ) and returns a one-hot vector ai of A categories using Gumbel-Softmax . AAE ’ s decoder APPLY takes the current state zi,0 and the action ai and reconstructs a successor state z∼i,1 ≈ zi,1 , acting as a progression function APPLY ( ai , zi,0 ) = z∼i,1 . APPLY is typically just called a “ model ” in model-based RL literature . While APPLY can be any network from the training standpoint , such a neural black-box function does not directly translates to a STRIPS action model , preventing efficient search with state-ofthe-art classical planner . Cube-Space AE ( Asai & Muise , 2020 ) addresses this issue by Back-ToLogit technique ( BTL Fig . 2 , bottom-left ) which modifies APPLY . Latent state transitions learned by BTL guarantees that the actions and the transitions satisfy the STRIPS state transition rule s′ = ( s \ DEL ( a ) ) ∪ ADD ( a ) , thus enabling a direct translation from neural network weights to PDDL modeling language . Details of the network , the translation method and the proof can be found in the appendix Sec . A.3 .

Summary: This paper addresses the problem of learning dynamics model directly from raw sensory inputs. The authors propose an unsupervised end-to-end model that can perform high-level tasks planning on raw observations. This work extends Asai et al. 2020, 2019 etc, and with improved symbol generation and lifted PDDL. The authors follow the experimental setup as seen in prior work, where three artificial environments (blocksworld, MNIST 8-puzzle, and sokoban) are used for planning.