Generating Plannable Lifted Action Models for Visually Generated Logical Predicates
Authors: Masataro Asai
Venue:  (2021)
Keywords: Object-Centric Representation
Planning
Discrete VAE

Abstract: We propose FOSAE++, an unsupervised end-to-end neural system that generates a compact discrete state transition model (dynamics / action model) from raw visual observations. Our representation can be exported to Planning Domain Description Language (PDDL), allowing symbolic state-of-the-art classical planners to perform high-level task planning on raw observations. FOSAE++ expresses states and actions in First Order Logic (FOL), a superset of so-called object-centric representation. It is the first unsupervised neural system that fully supports FOL in PDDL action modeling, while existing systems are limited to continuous, propositional, or property-based representations, and/or require manually labeled input for actions/predicates/propositions.

Under review as a conference paper at ICLR 2021
GROUNDING PLANNABLE LIFTED ACTION MODELS FOR VISUALLY GROUNDED LOGICAL PREDICATES
Anonymous authors Paper under double-blind review
ABSTRACT
We propose FOSAE+,, an unsupervised end- end-to-end neural system that generates a compact discrete state transition model (dynamics / action model) from raw visual observations. Our representation can be exported to Planning Domain Description Language (PDDL), allowing symbolic state-of-the-art classical planners per- form high-level task planning on raw observations. FOSAE++ expresses states and actions First-Order Logic (FOL), superset of so-called object-centric rep- resentation. It is the first unsupervised neural system that fully supports FOL in PDDL action modeling, while existing systems are limited to continuous, propo- sitional, or property-based representations, and/or require manually labeled input.
INTRODUCTION
Learning high-level symbolic transition model of environment from raw input (e.g., images) is a major challenge in the integration of connectionism and symbolism. Doing so without manually defined symbols particularly difficult it requires solving both the Symbol Grounding (Harnad
1990 Taddeo & Floridi 2005 Steels 2008 and the Action Model Leamiin/Acquisttionn problem.
Recently, seminal work by Asai & Fukunaga (2018) Latplan) that learns discrete planning mod- els from images has opened the door to applying symbolic Classical Planning systems to wide variety of raw, noisy data. Latplan uses discrete variational autoencoders to generate propositional latent states and its dynamics (action model) directly from images. Unlike existing work, which re- quires several machine learning pipelines (SVM/decision trees) and labeled inputs (e.g., a sequence of high-level options) (Konidaris et al. 2014), Latplan is end-to-end unsupervised neural net- work that requires no manually labeled inputs. Numerous extensions and enhancements have been proposed: Causal InfoGAN (Kurutach et al.) 2018) instead uses GAN framework obtain propo- sitional representations. Latplan's 's representation was shown to be compatible with symbolic Goal Recognition (Amado al. 2018). First-Order State AutoEncoder (Asai 2019 FOSAE) extends Latplan to generate predicate symbols. Cube-Space AutoEncoder (Asai & Muise 2020 CSAE) reg- ularized the latent space a particular form which directly exports to learned propositional PDDL model (Fikes et al. 1972). Discrete Sequential Application of Words (DSAW) learns a plannable propositional word embedding from natural language corpus (Asai & Tang 2020).
In this paper, we obtain a lifted action model expressed in First-Order Logic (FOL), which is a su- perset of object-centric (property-based) representation that Machine Learning community recently began to pay attention td but has long been the central focus of the broader AI community, In propositional action models, the environment representation not transfer to a different or dynamically changing environment with varying number of ob- jects. contrast, lifted FOL representations are generalized over objects and environments, as we demonstrate in Blocksworld with different number of blocks, or Sokoban with different map sizes. We propose Lifted First- Order Space AutoEncoder (FOSAE++) neuro-symbe neuro-symbolic architecture, which learns a lifted PDDL action model by integrating and extending the FOSAE, the CSAE and the Neural Logic Machine (Dong et al. 2019 NLM) architectures.
fixed-sized binary array and does
The overall task of our system is illustrated in Fig. I The system takes transition dataset contain- ing a set of pairs of raw observations which are single time step away. Each observation consists of multiple visual segmentations of the objects. It leams a lifted action model of the environment
c.g. ICML workshop on Object-Oriented Learning https: //ol workshop. www.kksoo..gthhti. github. io/

---

Under review as a conference paper at ICLR 2021
by generating the symbols and emits PDDL (Haslum al. planning systems.
2019) encoding for state-of offte--ofliee Sate-offtheearr tt-hhe-rrt
Contribution Table contains laxonomy of existing model acquisition systems in chronologi- cal order. FOSAE++ is the first system that satisfies all features readily available in symbolic action model acquisition systems, while not relying on human hurannderived symbols. FOSAE+ generates named symbols by itself Effectively addressing the long-standing Knowledge Acquisition bottle- neck (Cullen & Bryman 1988) and the Symbol Grounding problem, showing future direction for high-level symbolic autonomy.
0
1 time step
0
observations & training human
State description
Description of the transition
(cation move
Lifted action rule in PDDL
(clear
'parameter (Tblock ftrm Ptc)
(on 2)
?block
reffects land (on rblock Pto)
(clear
Pfrom 6,
(not 7block Mfam))
(not (clear Tta]
(on 4)...
?to =
(clear Mromr])
FOSAE++
(P
P 2)
generated
[ag 1
(aation
:parameter
(observation segmentation pairs)
Transition dataset
(i 5)
4)
predicate
symbols
7x,
2x, 6, generated
(not (p, 7xy])
(not (ps 7x, ?xg])
(p, 7x,
(p: 3)...
?Xg 2 action symbol
hxsll
Figure 1: FOSAE++'s learning task compared to the action modeling by humans. All symbols are anonymous (GENSYM'd due to the unsupervised nature. The produced PDDL files are then used to reason in the environment using classical planners.
Symbol Generation
Compatibility
Propo- Multi-arity predicates
End-to- End NN
Scarch algorithms
Propositional
Lifted
Action
sitional
PDDL PDDL
Symholic approaches'
manual
manual
manual
no
yes
yes
yes yes
Konidaris et al. 2014
manual" yes unused yes
yes unused
Lappann Asar & Fukunaga, 2018
yes
yes no
CausalInfoGAN (Kurutach et al.
2018
yes unused unused yes yes manual yes yes
incomplete"
n10
FOSAE (Asai 2019
nl
James et al. 20206
incomplete
manual" unused yes
yes
incomplete"
Cübe-Space ABT Asai & Muise 2020
yes
yes
yes
yes no
FOSAE++
yes yes yes
yes
yes
yes yes
Table I: Taxonomy of action model acquisition systems in chronological order.
PRELIMINARIES AND BACKGROUND
We denote a multi-dimennional array (tensor) in bold and its elements with a subscript (e.g., x I N D2 E M,,, an integer range I < m by n..m, concatenation of tensors and b in the last axis by a; b, and the i-th data point dataset by a superscript which we may omit for clarity. We use the same symbol for set and its size (e-g., S, and not (S)) to avoid the clutter. Finally, [0. C R. We assume background knowledge of discrete VAEs with continuous relaxations (BC) (Jang ct al. 2017 Maddison et al. 2017). Their activations are denoted as GS and BC, respectively.
(included in the appendix Sec. A. such as Gumbel-Softmax (GS) and Binary-
LIFTED STRIPS/PDDL PLANNING
Planning Domain Description Language (PDDL) is modeling language for Lifted STRIPS
planning formalism (Fies et al. 1972) and its extensions (Haslum et al. 2019). Let F(T) be
Yang al. 2007
Cresswell et 2013 Aineto et 2018 Zhuo al. 2019 Cresswell & Gregory
2011 Mouráo et al. 2012 Zhuo & Kambhampati 2013)
(004) requires sequences of high-level options learn from, such [move, move, interact. ... in Playroom domain. Causal InfoGAN cannot deterministically enumerate all successors (a requirement for search completeness) due to the lack of action symbols and should sample the successors. James et al] (2020b)'s PDDL output is limited to unary predicates properties of objects, thus cannot model the interactions between objects. Also, it requires sequences of high-level options such WalkToltem, AttachBlock, WalkToltem, in the Minecraft domain.
Konidaris
2

---

Under review as a conference paper at ICLR 2021
formula consisting of logical operations (A,-) and a set of terms T. For example, when = (have(1. (I. f full(I),, then (.ve(I.food) A -ful(I) E F(T). We denote a lifted STRIPS planning problem a 5-tuple P.,P,A,I, O is set of objects ( food), P is a set of predicates (9 full(z)), and A is a set of lifted actions ( eat). Fach predicate E P has an arity tp 2 0. Predicates are instantiatedlgroundee into propositions P(O) UpEP (1p) x O x .P. x xO, such as have(1, food). A state 8 S P(O) represents truth assignments to the propositions, e.g. = {have(1, food) represents have(1, food) = T. We can also represent it as bitvector of size Ep Otp
Each lifted action a(X) E A has an arity and parameters X (x1, Tea) such as eat(21, 12). Lifted actions are instantiated into ground actions A(O) UuEa ({a) x 0 x 49 x Q), such as eat(),food). a(X) ) is a 3-tuple (PRE(a), ADD(a), (EI. where ADD((), (a)(a) E F(P(X)) are preconditions. add-effects, and delete-effects: e.g., eat(a1, T2) ((have(z1,
(full(z1)): (have(æ1 xz))) The semantics of these three elements are as follows: A ground action a" E A(O) is applicable when state s satisfies PRE(a"), i.e., PRE(at) C. 8, and applying an action at to yields a new successor state (s)) = (s \. (aT( U ADD(at), e.g. eat(I.food) = "I can eat a food when have one, and if eat one I am full but the food is gone." " Finally, I,G S P(O) are the initial state and a goal condition, respectively. The task of classical planning is to find plan (a),...,,d) .aa))) which satisfies at, o....ca((1) G.
2.2 NEURAL PROPOSITIONAL//ATIIO SYMBOL GENERATION WITH LATPLAN
Latplan is framework for cammain-independeee image-based classical planning (Asai & Fukunaga 2018). It learns a propositional state representation and transition rules entirely from image- observations of the environment with discrete VAEs and solves the problem using classical planner. Latplan is trained on a Iransition input Tr: a set of pairs of raw data randomly sampled from the environment. The i-th transition (o o E Tr is pair of observations made before and after an unknown high-level action is performed. Once trained, Latplan can process a planning input (o', )), a pair of raw images corresponding to an initial and goal state of the environment. The output of Latplan is a data sequence representing the plan execution (o!,...o)) that reaches of from o While the original paper used an image-based implementation, conceptually any form of temporal data is viable for this methodology, e.g., an NLP corpus (Asai & Tang 2020).
The latest Latplan Asai & Muise (2020) has a training phase and a planning phase. In the training phase, it trains an end-to-end neural network called Cube-Space AutoEncoder (CSAB) on Tr (Fig 2 top left). CSAE is a variational autoencoder modeled by binary and categorical random variables each representing the propositional states and the actions in the classical planning. The dynamics modeled by these actions directly compiles into a PDDL model. The network combines Binary- Concrete VAE to produce binary state representation, and a Gumbel-Softmax VAE to produce
Space AE
O
i,0
i..
Back-to-Logit (BTL)
0
1
apply
continuous repr.
0
1,1
i,1
discrete repr.
Normalization
Batch
BN
sumo
BinConcrete
zi.1
MLP
action
apply
1,1
a
eeffect
discrete repr.
i
Z
loss
Simplified First Order State AE (SimpleFOSAE)
Raw observation
z/1
Ō
decode
@
(1,2)
@ (1,2)
(2,0)
8
@ (2.0)
Object detection
(0,0)
[0.0]
(preprocessing)
(Input) Object feature vectors
pointwise conv
Reconstruction
Figure 2: Cube-Space AutoEncoder, Back-To- Togio and First-Order State AutoEncoder.
3

---

Under review as a conference paper at ICLR 2021
categorical bottleneck layer which assigns categorical label to each input. Let o and of be pair of observed states in a transition, x" and z' be the corresponding binary latent states, and a be the one-hot vector that represents discrete action label assigned to the transition. CSAE is variational autoencoder network that can be formalized as follows:
(encoder)
zz ENCODE (o"), ENCODE(O') E B
(action assignment/elustering))
ACTION(2 =0 1,²) IB
(learning the dynamics)
1 = APPLY a) E B
=1 DECODE(Z"), DECODE(Z"), DECODE(*")
(decoder)
where ENCODE, DECODE, ACTION, APPLY are arbitrary multilayer perceptrons. The outputs of ENCODE and APPLY are activated by Binary Concerete, and the output of ACTION activated by Gumbel-Softmax of A categories (hyperparameter)) Assuming a certain set of prior distributions, lower bound (ELBO) of the log likelihood of observing a pair of states (o", o') can be derived follows (Appendix A.3.3)
log 1 o') 2 - DDKL(q(a 20, = 21))((((a==) 2²)) - (a =a ='))la(='(o)) "DkI(g(='lo')) (o>>)>>>>>>)))))) DKL (a(=)ol lo))))((=))) + log 2²) logp(o' 2,,22
After the training, it generates a propositional classical planning problem (z', x0 from planning input (o', o and export it into PDDL files together with the action model obtained in (2), which are then solved by Fast Downward (Helmert [2066), an optimized C++-based solver independent from the neural network. Finally, obtains a step-by-step, the plan execution by decoding the intermediate states of the plan into images.
comprehesimee visualization of
The A-dimensional one-hot categorical variable at in the network performs a clustering on the state transitions with the maximum number of clusters A specified as a hyperparameter. The cluster ID used as its action symbol. The clustering is performed by its encoder, ACTION(2 = = a', which takes propositional state pair (z**0 xz11 and returns one-hot vector a' of A categories using Gumbel-Softmax, AAE's decoder APPLY takes the current state z and the action a' and reconstructs a successor state zi.1 x 2 acting as a progression function APPLY(a', z(0)) = gi.
APPLY is typically just called "model" in model-based RL literature.
While APPLY can be any network from the training standpoint, such a neural black-box function does not directly translates to STRIPS action model, preventing efficient search with state-of- the-art classical planner. Cube-Space AE (Asai & Muise (020)) addresses this issue by Back-To- Logit technique (BTL Fig. bottom-left) which modifies APPLY. Latent state transitions learned by BTL guarantees that the actions and the transitions satisfy the STRIPS state transition rule s' (s \ (a) ADD(a), thus enabling direct translation from neural network weights PDDL modeling language. Details of the network, the translation method and the proof can be found in the appendix A.3 A.3
2.3 PREDICATE SYMBOL GENERATION WITH SIMPLE FIRST-ORDER STATE AUTOENCODER
First-Order State AutoEncoder (Asail 2019 FOSAE, Fig bottom) is autoencoder that takes a set of object feature vectors, identifies its latent predicate representation, then outputs a recon- struction of the input. Unlike prior work relational modeling (Santoro et al. 2017 Zambaldi ct al., 2019; Battagla et al.] 2018), this system obtains discrete logical predicates compatible with symbolic systems. The input is similar to the setting of Ugur & Piater (2015) and James et al. (2006)) Each feature vector can represent each object in the environment in an arbitrary complex manner. In this paper, we use segmented pixels combined with bounding box information (z, y and width, height). Let O,, E Rf be an F /-ddimensional feature vector representing each object and (o).... 00) E ROx F be the input matrix representing the set of O objects.
FOSAE generates multi-arity latent representation. Assume we represent the input with a com- bination of predicates of different arities. We denote set of predicates of arity n as P/n (Prolog notation) and its propositions as P/n(O). We denote the binary tensor representation of P/n(O) ".i,...,2/N]]] where N is the largest arity. The total size of its latent space is >==N O O* P/n. To convert the input into unary
as z/n E BOxx. xxxxxxxxPxp P/n. and the latent space is a tuple
4

---

Under review as a conference paper at ICLR 2021
predicates P/L, we apply a ID pointwise convolutional filter fi of P/1 output features the objects and activate it by BinaryConcrete for discretization, i.c., x x/1, (c(f((o)) B Sim- ilarly, for binary relationships P/2, we apply a filter f2 over concatenated pairs of objects, i.c. z/2;; = ( E BP/2, We can extend this framework to an arbitrary arity n. The multi- arity latent tensors (z/1. 1e/1..... ..... z z/N) can be flattened, concatenated, then decoded to the reconstruc- tion Ô E R by an arbitrary decoder network (MLP in the original paper).
There is a potential exponential blowup due to O parameter combinations. This problem is ad- dressed by attentions the original work (Asai, 2019). but we omit this feature this paper for simplicity and due relatively small N(< 4). We call our variant as SimpleFOSAE.
2.4 NEURAL LOGIC MACHINE (NLM)
The NLM (Dong et al. 2019) is neural Inductive Logic Programming (ILP) (Muggleton 1991) system whose inputs and outputs are hand-crafted binary tensors representing propositional ground- the multi-arity representation (z/1...., z/N)
ings of FOL statements. These binary tensors identical to the latent space of SimpleFOSAE, NLM provides a key feature that is necessary for learning lifted First-Order Logic action model: Invariance to the number and the order of objects in the multi-arity representation. The latter implies permutation equivariance, i.c., when the first n axes of each input z/n E B P/r are reordered by a permutation т, the output axes are also reordered by π. more detailed explanation of NLM can be found in the appendix Sec. Sec.A..2 A.2
An NLM contains N COMPOSE operations which are applied to the corresponding elements of (z/1,-.. ... ,,,.:,,,,NNN...... Each COMPOSE is denoted as COMPOSEq..<2. n) E BO where Q is a constant that specifies the number of output features, and is nonlinearlity. An NLM output is denoted as NLMQ, (z) = (COMPOSEQ.d (z, 1), (ZMPOSEq.
3 LIFTING THE ACTION MODEL
In order obtain a lifted action model in an unsupervised setting, there are three requirements: (1) white-box box action model which trivially convertible to STRIPS formalism, (2) invariance to the number/order of objects, (3) unsupervised generation of multi-arity predicate symbols. To our knowledge, no existing systems fulfill all requirements: FOSSAE lacks and 2, CSAE lacks 2 and and NLM lacks 1 and 3 (designed for hand-crafted symbolic data).
(FOSAE encoder)
0.0
f11 ENCODE (D)) ENCODE(O
(action parameters selector in NLM)
x² PARAMS 1,0 2(11
(extract the parameter-bound subspace)
4.
.1 BIND x²), BIND(2 ,x)
(unsupervised action assignment)
a' ACTION(2] 1.0 (2+ 2411
(learning the bounded dynamics)
g(1 = APPLY( 6.0
(reflection the global dynamics)
211 200 UNBIND 2100 xx)) UNBIND(S x²)
(decoder in NLM) 04.0 o1.1 21.1
DECODE(2*0) (2)), DECODE 2d)1) (EEODDEEE
Loss: f 0..0) E(o''1 (
+l(o'.1 aill 1 +((())))))) 2dd) E(Z)1 i1l 2 + regularization.
We now propose FOSAE+ whose overview is shown above, which addresses issues once. Its overall design follows the CSAE, which ENCODE the input, identifies the action a', APPLY-es the action, then DECODE s the results, uses FOSAE's encoder to generate multi-arity latent represen- tation, which is then consumed by NLMs used as a basic building block of other components. This architecture 's key element a new component PARAMS and a unique pair of operations called BIND and UNBIND, which intuitively reflects the structure of lifted action models. Suppose lifted action (move ?block ?from ?to) in Blocksworld (Fig. with effects such as (on ?block ?to). Since lifted model always refers the objects through its parameters such as ?to, it cannot affect
model
5

---

Under review as a conference paper at ICLR 2021
truth cube 2/3 (boolean tensor) a ternary predicate p(?x,y,,2)
binding for moveeffrom, itt) where (?from, ?to) = (c,4)
bind(z/3.(c.a))
c
?z
?2
praduct
Pfrom Tto
3
a b ?x
b by 0
Note: ordering has changed.
Figure 3: move(c,a) should not affect p's groundings with and d, therefore BIND limits the ground- ing to and c. move(c.a) and move(a.c) can then apply the same effect to each subspace due to the axis reordering. (Note: move(?block, ?from, ?to) is simplified to move ?to) for illustration.)
the objects that does not appear in the parameter list, e-g., action (move a b c) cannot affect (on c) because d & (a. b, c). We represent this restriction differentiable mutrix operations.
To implement this idea, we first added a new attention network æ = PARAMS( 2211 (æI,..., 2ta) which attends to the objects, essentially learning the parameters of the action. We assume all actions have the same number of parameters, and thus #a is a hyperparameter. Each pa- rameter D; is a one-hot hard attention vector in BO CE Mij = 1) activated by Gumbel-Softmax. This can later extract a specific object from the object feature matrix o by an inner product æ; RO - IR ). PARAMS has several NLM layers, ending with ILM We then extract its unary part of the results (-> BOx*)), transpose it (+ R*ax(), then apply #a-way Gumbel- Softmax of O categories (- BdoxO), As result, the output attends to #a objects in total.
We utilize the attentions in unique operations named BIND and UNBIND. Recall that the predicates in the effects can refer only to the specified action parameters. Therefore, we limit the dynamics to the objects attended by (Fig.) 3. We iteratively extracts the sub-axes of z/n attended by using matrix operations (with an abuse of notation) 2t/ (x)"²2/n z/n (B*axo (BB:xO)=o:::0PP)))
We call it BIND(=/n) (z/n, x), as it binds the parameters X to the values in objects O similar to the function call in typical programming language. It is also similar to applying numpy.take in all dimensions. Binding z/ z/n for all arities results in BIND( x) = (BIND(z/L, x)), -. BIND( x)). We also define UNBIND(Z)/1) x)) (x )" BOm.xx P/n which restores the original shape, but fills the cells with zeros for the objects not attended by x.
Example 1. For a moment let's ignore that we use a boolean tensor and let z/2 [[1, 2, 3). [4,5, 5, 6). [7, 8, 9]] with
[1,2,3] 3} which represents a two-arg function f(2 [[.01, [.98, 001, O1]], we obtain 2t/2 ==(z/2)=² 2 [5., 4.j, [211, 1.1]] ~ [|5, 4), [2, 1]], appreximating the subspace extraction. To un- bind it, x (2/2))= [1.1, 2.1, 03), [3.9, 4.9, .09], [.05, .01] R [1, 2, 0), [4,5, 0), 0,
z/22.1 4 etc. When we bind z/2 2 (2, 1) by
After the subspace extraction with BIND, model inside the subspace. Notice that ACTION and APPLY no longer directly refers to each object o by the index because BIND already resolves the index reference. They take flattened binary vectors of size Σ, (#a)" " P/n as the input/ / output.
APPLY and ACTION find the dynamics / action
In order to reflect the changes applied to the bounded representation the whole representation, UNBIND: g-1 z UNBIND( 1.0) UNBIND 2 UNBIND is unique in that the unattended axes have near-O values (0 the limit of Gumbell-offtnn annealing in PARAMS). Therefore, the propositions not bound by x retain their values from z1.0
Finally, we replace SimpleFOSAE's MLP decoder with NLM layers. To match the output shape with the input o E R the final layer has F features and we use only the unary part of the result tuple, N ()/1 COMPOSE(- (,, 1, F, The total loss is the sum of i g0.), E(oi.) , (( g+1) ) ((2(1 111, and {x 1.1 w
plus the variational losses for discrete VAEs. Hyperpa- rameters, choice of loss functions and the tuning process are detailed in the appendix Sec.
The number of parameters in FOSAE++ is not affected by the number of objects, since essentially each NLM layer performs 1D pointwise convolution over tuples of objects.

---

Under review as a conference paper at ICLR 2021
Objecss
represented object
+
A
a
5
6
1
7
4
5
2
Figure 4: (top) State transitions in Blocksworid and 8- 8-puzzle, and the initial/goal state of a Soko- ban problem. In Sokoban and 8-puzzle, each tile is represented as an object. (bottom) Example rendering of the ground-truth / reconstruction errors. FOSAE++ reconstructs rectangular images and bounding boxes, then draw them on black canvas. Due to the object-based input, does not produce the background. The depicted Sokoban reconstruction consists of 63 tile objects.
35
Figure 5: (Synopses:) The networks are trained on X objects, then tested on Y objects. (Top: Interpolation) 8-Puzzle: X = 9, Y E (3, 5, 6, 7]. Sokoban: X= 46, Y E {13. 18, 23, 32). Blocksworld: X 6, Y E (3, 5).. (Bottom: Extrapolation) Sokoban: 46 (problem pO), Y E {64, 41, 64, 43) (p01-p04). Blocksworld: X = = Y (4,5,
EXPERIMENTS
We prepared 3 artificial environments: Photo-realistic Blocksworld (Asai 2018) renders an RGB Blocksworld image with Blender 3D engine and extracts the bounding boxes and the image patches of each object. MNIST 8-Puzzle (Asai & Fukunaga 2018) is 42x42 pixel, monochrome image- based version of 8-Puzzle. Tiles contain hand-written digits (0-9) from the MNIST database (LeCun et al.) 1998), and valid moves swap the "0" tile with neighboring tile, i.e., the "O"" serves as the "blank" tile in the classic 8-Puzzle. Sokoban (Culberson) 1998) is a PSPACE hard puzzle domain whose visualization was obtained from the PDDLGym library Siiver & Chitnis 2020). In all datasets, segmentation is already provided by a domain-specific code, but in principle it can be replaced by the output of object-detectors such as YOLO (Redmon & Farhadi 2018). The detail of the data preparation is available in the appendix Sec. D Fig 4ig.4ss some comparisons between the input i. and the reconstruction 00. In Sokoban, FOSAE++ generates up to 63 objects in a single scene.
4.1 INTERPOLATION/EXTRAPPO EPPOOOTTTIITAAAOAO TO VARYING, UNSEEN NUMBER OF OBJECTS
The key characteristics of the FOSAE++ is that once trained, its network structurally generalizes to the different number of objects. We demonstrate this performance in interpolation/eetrrpolatii tasks, where the former we randomly drop a certain number of objects from the input, and the latter, use environment with different distribution more objects. In both tasks, FOSAE++ shows an excellent reconstruction for a varying, untrained number of objects (Fig 5).
4.2 PLANNING EXPERIMENTS
Finally, Fast Downward Helmert (206)) planning system with LMeut Helmert & Domsh-
lak (2009) heuristics and A search confiurration on the PDDL domains generated by FOSAE++. For the planning succeed, FOSAE++ must be accurate not only about the direct reconstruction

---

Under review as a conference paper at ICLR 2021
04.0 of.l but also about the dynamics that predicts the successor state 211 and its reconstruction g4111
Due to the time constraint, we only performed the experiments for 8-Puzzle as of now. From the fixed goal state (the solved state of the puzzle), we performed a domain-specific Dijkstra search and sampled the initial states optimally L-steps away. Each initial/gool states are rendered, cropped into object-based based imput, then encoded by FOSAE++ to produce the symbolic initial/goal states. The planning results visualized and manually inspected//aliddtee by us. The symbolic classical planner correctly produced a solution which can be visualized into correct state transitions. See Sec. for the visualization of all 20 instances.
8- B-Puzzle
(L 5) 10/10
(L = 10)
10/10
203
Table 2: Planning experiments (solved instances/ total) and an example visualization obtained from the system (L 10).
RELATED WORK
James et al. (2020a b) build existing work (Konidaris et al., 2014 2015 2018) to find an ego- centric logical representation that invariant to the state of the observer in a complex environment, object-centric, property-basel logical representation. Both representations of First-Order Logic representation where all predicates are unary. These approaches also require a training input that already contains action symbols (e.g., toggle-door) Andersen & Konidaris (2017) obtains an MDP-based PPDDL (Probabilistic PDDL) model using Active Learning. Incorporating Active Learning in FOSAE++ for self data collection is future work.
special cases
Huang et al. 2019) reported that direct discretization approach performed worse than an approach that plans in the probabilistic representation obtained by neural networks. However, the question of continuous versus discrete is not conclusive. They used naive rounding-basse may cause significant amount of information loss compared to the state-of-the-art discrete varia- tional autoencoders. Furthermore, they rely on mappings from raw inputs to hand-coded symbolic representations that require supervised training.
discretization which
Object-based input we used can be obtained from state-of siiie--f-theeaae object-recognitin methods such as YOLO (Redmon & Farhadi 2018). More recent systems (Greff et al. 2019 Engeleke et al.] 2020) can handle shapes not limited to rectangles, and can be trained unsupervised. Connecting our network with these systems is an exciting avenue of future work.
Eslami al. (2016) and Xu et al. (2019) proposed an attention basdd and Bayesian approach to the problem variable-sized object reconstruction. Their work do not address the state dynamics and is orthogonal to our work. The primary difference from our approach is that they studied on sequentially storing and retrieving a variable amount of information into/from a fixed-sized array, while store them in latent representation of the corresponding size.
6 CONCLUSION
We proposed a first fully neural First-Order Logic Action Model Acquisition system that success- fully produces lifted PDDL planning model from noisy, segmented images alone. The system generates three types of symbols (action, predicate. and propositional) without manual labeling, and provides STRIPS-compate objects, thus can extrapolate to different and dynamically changing environments. We demonstrated that the state-of-the-art classical planner can produce correct solutions (as theoretically guaranteed) only with the interactions with the learned model. Note that the classical planner we used has no learning components. Our results partially support the Physical Symbol Systems Hypothesis by Newell & Simon (1976): In word, formal high-level representation learned from the raw in- puts is sufficient for intelligent actions. Future work includes extensions to the formalisms with higher expressivity, including Probabilistic PDDL (Younes & Littman 2004). Axioms (Thiébaux
explanations of actions. The learned results are generalized over the
8

---

Under review as a conference paper at ICLR 2021
al. 2005 ADL (Pednault 1987). numeric planning (Fox & Long 2006), or Hierarchical Task
Networks Erol et al. 1994 Nau et al. 2003 Nejati et al. 2006).
REFERENCES
Diego Aineto, Sergio Jiménez, and Eva Onaindia. Learning STRIPS Action Models with Clas- sical Planning. In Proc. of the International Conference Automated Planning and Schedul- ing(ICAPS), 2018.
Vidal Alcázar, Daniel Borrajo, Susana Fernández, and Raquel Fuentetaja. Revisiting Regression in Planning. Proc. of International Joint Conference on Artificial Intelligence (IJCAT), 2013.
Leonardo Amado, Ramon Fraga Pereira, Joao Aires, Mauricio Magnaguagno, Roger Granada, and Felipe Meneguzzi. Goal Recognition in Latent Space. 2018.
Garrett Andersen and George Konidaris. Active Exploration for Learning Symbolic Representations. In Advances in Neural Information Processing Systems, pp. 5009-5019, 2017.
Masataro Asai. Photo-Realistic Blocksworld Dataset. arXiv:1812.01818 2018. URL http://
arxiv. wtt./bs//a////// 01818
Masataro Asai. Unsupervised Grounding Plannable First-Order Logic Representation from Im- ages. In Proc. of the International Conference Automated Planning and Scheduling(CAAPS), July 2019.
Masataro Asai and Alex Fukunaga. Classical Planning in Deep Latent Space: Bridg- ing the Sobsymbolic-Symboll Boundary. Proc. of AAAI Conference on Artificial In- telligence, 2018. URL https:/ http://www... aaai. www///s.....oooooooooooooooooooe index. phpp paper/view/11 16302
Masataro Asai and Hiroshi Kajino. Towards Stable Symbol Grounding with Pero-Suppressed State AutoEncoder. In Proc. of the International Conference Automated Planning and Schedul- ing(ICAPS), July 2019.
Masataro Asai and Christian Muise. Learning Neural-Symbolie Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS). In Proc. of International Joint Conference on Artificial Intelligence (ICCAI). 2020.
/ /ttpxvaiiiv/..////////aa//aaaav/ org/ abss/2004 12850
Masataro Asai and Zilu Tang. Discrete Word Embedding for Logical Natural Language Understand- ing, arXiv preprint arXiv: 1et..1116660 2020.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Re- lational inductive biases, deep learning, and graph networks. arXiv preprint crXiv:1806.01261, 2018.
Stephen Cresswell and Peter Gregory. Generalised Domain Model Acquisition from Action Traces. In Proc. of the International Conference Automated Planning and Scheduling//CAPP), 2011. URL http:// org. ocs/ /atessiindde index. http://cass/IIeeee///ppee//////
Stephen Cresswell, Thomas Leo McCluskey, and Margaret Mary West. Acquiring planning do- main models using LOCM. Knowledge Eng. Review, 28(2):195-11 2013. doi: 10.1017/
80269888912000422..
Joseph Culberson. Sokoban is PSPACE-complete. In Proceedings in Informatics 4, International Conference on Fun with Algorithms, pp. 65-76. Carleton Scientific, June 1998.
J Cullen and A Bryman. The Knowledge Acquisition Bottleneck: Time for Reassessment? Expert Systems, 5(3), August 1988.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural Logic Machines. In Proc. of the International Conference on Learning Representations, 2019. URL
net wtt.:e/ooooomomidllll//----- http://ggeeeeeeeeeeeeeeee//ooooo

---

Under review as a conference paper at ICLR 2021
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Genera- tive Scene Inference and Sampling with Object-Centrid Latent Representations. In Proc. of the International Conference Learning Representations, 2020.
Kutluhan Erol, James Hendler, and Dana Nau. HTN Planning: Complexity and Expressivity. In Proc. of AAAI Conference Artificial Intelligence, 1994.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hin- ton, et al. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Richard E Fikes, Peter E. Hart, and Nils J. Nilsson. Learning and Executing Generalized Robot Plans. Artificial Intelligence, 311-3)255 1972. doi: 10.11016000433702222)00000000000 doi /ttp:/o.......0111111//////1// (20000000107702
URL
http:/
Maria Fox and Derek Long. Modelling mixed discrete-comtinuuus domains for planning. Journal of Artificial Intelligence Research, 27:235-299 2277--29,229 2006.
Kunihiko Fukushima. Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position. Biological Cybernetics, 36(4), 1980.
Martin Gebser, Benjamin Kaufmann, Roland Kaminski, Max Ostrowski, Torsten Schaub, and Mar- ius Schneider. Potassco: The Potsdam answer set solving collection. Al Communications, 24(2):
107-124, 2011.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In Proc. of the International Conference on Machine Learning. pp. 2424-2433, 2019.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335- 346, 1990.
Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, and Christian Muise. An Introduction to the Planning Domain Definition Language, volume 13. Morgan & Claypool Publishers, 2019.
Malte Helmert. The Fast Downward Planning System. Artif. Intell. Res.(JARR)) 26:191-246, 2006.
URL http: /wwww//wwww///mmc//////////.ww aaai wttp://ppg//////////////.a// http:///w.1///AR---- 260
Malte Helmert and Carmel Domshlak. Landmarks, Critical Paths and Abstractions: What's the Dif- ference Anyway? In Proc. of the International Conference on Automated Planning and Schedul- ing(ICAPS), 2009.
//wp://///wwwww... aaai org// /tlpg//cs//// www.occnndeeeee http://nexxx.pp//IPP.... wwwp:ICCAP//CC.... ICAASS// paper/viewPDER Inteennetettttta..ee wttpstttiiiaa/33//110000
De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, and Juan Carlos Niebles. Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning. In Proc. of IEEE International Workshop on Intelligent Robots and Systems (IROS), pp. 2635-2642. IEEE, 2019.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. of the International Conference Machine Learning, pp. 448-456 2015. URL htl.http:/// http://hh.....roooooeeeeeo/e//rp
html
Steven James, Benjamin Rosman, and George Konidaris. Learning Portable Representations for High-Level Planning. In Proc. of the International Conference on Machine Learning, 2020a.
Steven James, Benjamin Rosman, and George Konidaris. Learning Object-Centric Representations for High-Level Planning in Minecraft. In Proceedings of Workshop on Object-Oriented Learning at ICMI., 2020b.
Eric Jang, Shixiang Gu. and Ben Poole. Categorical Reparameterization with Gumbel-Softma In Proc. of the International Conference Learning Representations, 2017. 2017.
10

---

Under review as a conference paper at ICLR 2021
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei. C Lawrence Zitnick, and Ross Girshick. CLEVR: Diagnostic Dataset for Compositional Language and Elementary Vi- sual Reasoning. In Proc. of IEEE Conference Computer Vision and Pattern Recognition, pp- 1988-1997. IREE, 2017.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proc. of the Interna- tional Conference on Learning Reprexentations, 2013.
George Konidaris, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Constructing Symbolic Rep- resentations for High-Levee Planning. In Proc. of AAAI Conference on Artificial Intelligence, pp. 1932 1938, 2014, URL httpttt::// /tt////wwww.. aaai org/ www./r//g..ooooooooooooooo//eoee ocs www./iinneedeee.. wntp:.ndxxxppp. wwtpp//wwppppp/////aaaaaae
paper/view 8424
George Konidaris, Leslie Pack Kaelbling, and Tomás Lozano-Pérez, Symbol Acquisition for Proba- bilistic High-Level Planning. In Proc. of International Joint Conference on Artificial Intelligence (IJCAI), pp. 3619-3627, 2015. URL
George Konidaris, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. From Skills to Symbols: Learn- ing Symbolic Representations for Abstract High-Level Plamning. Artif. Intell. Res.(JAIR) 61: 215-289, 2018. doi: 10. 10.1613/jair.5555 https:/::
org 10. 1613 jair 5575
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, and Pieter Abbeel. Learning Plannable Representations with Causal InfoGAN, In Advances in Neural Information Processing Systems, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- -aased Learning Applied to Document Recognition. Proc. of the IEEE, 86(11):2278-23)) 2324, 1998.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In Proc. of the International Conference on Learning Representations, 2017.
Kira Mourão, Luke Zettlemoyer, Ronald P. Petrick, and Mark Steedman. Learning
STRIPS Operators from Noisy and Incomplete Observations. In Pruc. of the Interna- tional Conference on Uncertainty in Artificial Intelligence, pp. 614-623, 2012. URL
https: http::/dddppttttt.oooooo http:://wabbbl......aaaa//aaaaas 2tarticle 1d-23224proeeding 1d-28
jap 2mmnu= &amnu=
Stephen Muggleton. Inductive Logic Programming. New generation computing, 8(4):295-318
1991.
Vinod Nair and Geoffrey E Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proc. of the International Conference on Machine Learning, 2010.
Dana S Nau, Tsz Chiu Au, Okhtay Ilghami. Ugur Kuter, J William Murdock, Dan Wu, and Fusun Yaman. SHOP2: An HTN Planning System. J. Artif. Intell. Res.(JAIR), 20:379-404, 2003.
Negin Nejati, Pat Langley, and Tolga Konik. Learning Hierarchical Task Networks by Observation. In Proc. of the International Conference on Machine Learning, 2006.
Allen Newell and Herbert A. Simon. Computer Science Empirical Inquiry: Symbols and Search. Commun. ACM, 19(3): 13--126, 1976. doi: 10.1145/360018.330 330000002 URL TRLhtttp://do: acm.
360022
Charles Payan. On the Chromatic Number of Cube-Like Graphs. Discrete mathematics, 103(3), 1992.
Edwin PD Pednault. Formulating Multiagent, Dynamic- World Problems in the Classical Planning Framework. In Reasoning about actions & plans, pp. 47-82. Elsevier, 1987.
Joseph Redmon and Ali Farhadi. arXiv:1804.000 02767, 2018.
Yolov3: An incremental improvement. arXiv preprint
11

---

Under review as a conference paper at ICLR 2021
Raymond Reiter. On Closed World Data Bases. In Readings in Artificial Intelligence, pp. 119-140. Elsevier, 1981.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillierap. A Simple Neural Network Module for Relational Reasoning. In Advances in Neural Information Processing Systems, pp. 4967-4976, 2017.
Tom Silver and Rohan Chitnis PDDLGym: Gym Environments from PDDL Problems, 2020.
Luc Steels. The Symbol Grounding Problem has been Solved. So What's Next? In Manuel de Vega, Arthur Glenberg, and Arthur Graesser (eds.). Symbols and Embodiment. Oxford University Press, 2008.
Mariarosaria Taddeo and Luciano Floridi. Solving the Symbol Grounding Problem: A Critical Re- view of Fifteen Years of Research. Journal of Experimental & Theoretical Artificial Intelligence, 17(4):419-44, 2005.
Sylvie Thiébaux, Jörg Hoffmann, and Bernhard Nebel. Defense of PDDL Axioms. Artificial Intelligence, (68(1-2):38-6) 69, 2005.
Emre Ugur and Justus Piater. Bottom-up Learning of Object Categories, Action Effects and Logi- cal Rules: From Continuous Manipulative Exploration to Symbolic Planning. In Proc. of IEEE International Conference Robotics and Automaton (ICRA), pp. 2627-2633. IEEE, 2015.
Arash Vahdat, Evgeny Andriyash, and William Macready, DVAE#: Discrete variational autoen- coders with relaxed Boltzmann priors. In Advances in Neural Information Processing Systems, 2018a.
Arash Vahdat, William G Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny An- driyash. DVAE++: Discrete variational autoencoders with overlapping transformations. arXiv: 2018b.
Aaron van den Oord, Oriol Vinyals, et al. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, 2017.
Vadim G Vizing. The Chromatic Class of Multigraph. Cybernetics and Systems Analysis, 1(3), 1965.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric Cross Entropy for Robust Learning with Noisy Labels. In Proc, of IEEE Conference on Computer Vision and Pattern Recognition, pp. 322-330, 2019.
Kai Xu, Akash Srivastava, and Charles Sutton. Variational Russian Roulette for Deep Bayesian Nonparametrics. In International Conference on Machine Learning, pp. 6963-6972, 2019.
Qiang Yang, Kangheng Wu, and Yunfei Jiang. Learning Action Models from Plan Examples using Weighted MAX-SAT. Artificial Intelligence, 171(2-3) 2007. doi: 10.101 10.11066j.artinn.2200 2006. 11.005. URL http: doi http://w.......o111111111111////
2006 11. 005; http::/
db p. uni -et-rtriee.. www.iirrdeeeeeeeee httpr//r////////e///////////////
Häkan LS Younes and Michael L Littman. PPDDLI. 0: An Extension to PDDL for Expressing Planning Domains with Probabilistic Effects. Techn. Rep. CMU-CS-04-162, 2:99, 2004.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep Reinforcement Learning with Relational Inductive Biases. In Proc. of the International Conference on Learning Repre- sentations, 2019.
Hankz Hankui Zhuo and Subbarao Kambhampati. Action-Model Acquisition from Noisy Plan Traces. Proc. of International Joint Conference on Artificial Intelligence (IJCAI), 2013.
Hankz Hankui Zhuo, Jing Peng, and Subbarao Kambhampati. Learning Action Models from Disor- dered and Noisy Plan Traces. arXiv: 100iv:900..000 0900000000 2019.
12

---

Under review as a conference paper at ICLR 2021
A EXTENDED BACKGROUNDS
A.1 DISCRETE VARIATIONAL. AUTOENCODERS
Variational AutoEncoder (VAE) a framework for reconstructing an observation from a compact latent representation z that follows a certain prior distribution. Training is performed by maximizing the sum of the reconstruction loss and the KL. divergence between the latent random distribution ((2/2r) and the target distribution p(z), which gives lower bound for the likelihood p(z) (Kingma & Welling 2.13). While typically p(z) is a Normal distribution N (0, 1), Gumbel-Softman (GS) VAE (Jang et al. 2 and Binary-Concrete (BC) VAE (Maddison et al., 2017) use discrete, uniform categorical/Beman (em5 distribution, and further approximate it with a continuous relaxation by introducing temperature parameter T that is annealed down to 0.
We denote corresponding activation function in the latent space as GI( and BC((), where I and ( each represents logit vector and scalar. A latent vector [0, 1] of GS-VAE is computed from logit vector I E RC by z = GI) = SOFTMAX + GUMBEL (0, 1))/r), where C' is the number of categories, GUMBEL (0, 1)
log(- loog--ogooo 2) and UNIFORM (0, 1) E [0, 1|
A latent scalar = of BC- is computed from a logit scalar / E RR by = BC(() = SIGMOID + LOGISTIC(O, 1)) where LOGISTIC(O,) 1) log log(1 u) and UNIFORM(O, 1) R.
Both functions converge to discrete functions at the limit r - O: (I - arg max(I) (in one-hot representation) and BC( + STEP(!) = (1 < 0) ?0 1. Typically, VAE and BC-VAE contains multiple latent vectors latent scalars to model the complex input.
There are more complex variations such as VQVAE van den Oord al. (017), DVAE+ Vahdat et al.] (2018)), DVAE# Vahdat et al. (2018a). They may contribute to the stable performance, but we leave the task of faster/ easier training to the future work.
A.2 NEURAL LOGIC MACHINE
The NLM (Dong et al. system based on Order Logic and the Closed World Assumption (Reiter 1981). Given a set of base predicates grounded on set of objects, NLMs sequentially apply horn rules to draw further conclusions such as a property of or a relationship between objects. For example, in Blocksworld, based premise such as (on a b) for blocks b, NLMs can infer (not (clear b)). NLM has two unique features: (1) The ability to combine the predicates of different arities, and (2) size invariance & permutation equivaliance input objects, which is achieved by enumerating the permutations of input arguments.
2019) is neural Inductive Logic Programming (ILP) (Muggleton, 1991>
NLM is designed work on hand-crafted binary tensors representing propositional ground- ings of FOL statements. The format of these binary tensors are the multi-arity representation (z/1.... ,, =/N) identical the latent space of SimpleFOSAE.
NLM is designed for a subset of First Order Logic where every statement is a horn rule, contains no function terms (such as function that returns an object), and all rules are applied between neighboring arities. With these assumptions, the statements fall in one of the three types below:
(expand) Vxazi p(X; rop) < p(X), (reduce) p(X) Jxap :(:pp)))
(compose) x(X) F (UPUPUPP//M(X)))
Here, P.P.P.q E P.P.P.Q (respectively) are predicates, X (II,...) is a sequence of parame- ters, F(T) is a formula consisting of (A, V, and T, (P (PUPUP) | is a set of predicates of the same arity as q, and π (xX) is a permutation of X, which used for composing the predicates with the different argument orders.
All three rules can be implemented as tensor operations (Fig 6). Given binary tensor 2/n of shape O...O P/n, expand copies the n-th axis to n + 1-th axis resulting in shape O P/n, and reduce takes the max of n-th axis resulting in a shape O.:!0 x P/n. While the original paper
13

---

Under review as a conference paper at ICLR 2021
pointwise
output f
(horizontal copying)
1D conv
z/0
expand
b aa ab ac
Permute (12 output)
z/1
b ba bb bc
aa Ca
c
ca cb cc
aa ba ca
ab
z/2
reduce (vertical max)
3x3x4 (0=3, (/22|=4)
ab bb cb
ac 3x3x(12 4)
ac bC cc
Figure 6: NLM uses EXPAND, REDUCE, PERMUTE tensor operations to combine predicates of dif- ferent arities while maintaining the object-ordering size invariance. (For matrices, PERMUTE is equivalent to transposition.)
proposed both min and max versions to represent V and E, with enough number of layers only one of them is necessary because Vr;p(.; .) = -3x; +
Finally the COMPOSE combines the information between the neiboring tensors z/n, n/ 2/ n+1- In order to use the information in the neighboring arities (P, P and with EXPAND 1- and REDUCE(Z/n-) (-> O.T.O x C where C = P/n + P/n-1 P/n+1). Next, a PERMUTE function enumerates and concatenates the results swapping the first axes in the tensor (-> 0. O x (In-C) It then applies n-D pointwise convolutional filter fn with Q output features (+ O. x Q). the actual implementation, this n-D pointwise filter is merely a ID convolution performed on matrix reshaped into O (In-C", it activated by any nonlinearity to obtain the final result, which we denote as COMPOSE(z, n, o). Formally, Vj E 1..m, Voj E 1..0,
P). the input concatenates 2
COMPOSE( (2 n. Q) or... = oJn (oo)) )) R
where II PERMUTE EXPPAND(Z/n-I) z/n; REDUCE(#/n+1)) E (O=0x(m.C))
An NLM contains N (the maximum arity) COMPOSE operation for the neighboring arities, with appropriately omitting both ends (0 and N + 1) from the concatenation. We denote the result ................................ COMPOSE(2, NN ())... This horizontal arity-wise compositions can layered vertically, allowing the composition of predicates whose arities differ more than 1 (e.g., 2 layers of NLM can combine unary and quaternary predicates).
as NL.MQ. (z) (COMPOSE(Z, 1, 1, a),
Two minor modification was made from the original paper. First, we use slightly different tensor shapes: For the notational conveniency, we use dimerccubedimensionanl tensors of shape BO..Ox P/n instead of the original formulation 18000-1x...0- n-xx//n which tries to reduce the size by disal- lowing the duplicates in the parameters. Our modification does not significantly affect the complex- ity of the representation because the original representation also has O(O") complexity.
Second, we do not use the nullary predicates z/O in order to disallow VAEs from encoding inviromeen--ppccffi information it.
14

---

Under review as a conference paper at ICLR 2021
A.3 LEARNING DISCRETE LATENT DYNAMICS USING BACK-TO-LOGIT
APPLY 200) is arbitrary MIP,, i.e., a neural black-box function that does not directly translates to STRIPS action model, preventing efficient search with state-of-the-art classical planner. Cube- Space AE (Asai & Muise 2020) addresses this issue by Back-To-Logi technique which replaces the MLP. Back-to-Logi places a so-called cube-like graph prior on the binary latent space / transitions. To understand the prior, the background of cube-like graph is necessary.
Back-to-Logit (BTL)
zi.
apply
continuous repr.
discrete
repr.
Noralizatiin
Batch
BNN
BC
BinConcrete
Z
MLP
Peffect
discrete repr.
Figure 7: Back- To- architecture (repost of Fig.
A.3.1 CUBE-LIKE GRAPHS AND ITS EQUIVALENCE TO STRIPS
Asai & Muise (2020) identified that state transition graphs of STRIPS planning problems form a graph class called directed cube-like graph (Payan 1992). A cube-like graph simple" undi- rected graph G(S, D) = (V. E) defined by sets $ and /. Each node v E V is a finite subset of S, i.e., S S. The set D is family of subsets of S, and for every edge
E, the symmetric difference d = v @ w (v w) U (w \ 2) must belong to D. For exam- ple, a unit cube is a cube-like graph because {(0, {x]),... (fv,2), {æ,, y. 2))}. D = {{z}, {2]}. The set-based representation can be al- ternatively represented as a bit-vector, e.g., V = {(0,0,0) (0.0, 1), (1.1,1)). We denote a one-to-one bit vector assignment as V + (0,
(u,
{z.y.2), = {0, (z).... {z, y, 2}}, E
In STRIPS modeling, we use directed version of this graph class. For every edge e = (v, E E, there is pair of sets d = dd+ (w \ v. w) E D which satisfies the asymmetric difference = (u u d+ It is immediately obvious that this graph class corresponds to the relationship between binary states and action effects in STRIPS, s' = (s ()a)))a) U ADD(()a. As a special case, in an undirected STRIPS planning problem where all actions are reversible (i.e., for any action such that & a(s). there is an action a that is able reverse the effect a (t)), the state space graph is equivalent to an undirected cube-like graph.
A.3.2 EDGE CUROMATIC NUMBER OF CUBE-LIKE GRAPHS AND THE NUMBER OF ACTION SCHEMA IN STRIPS
Edge coloring of cube-like graphs provides an intuitive understanding of the action schema express- ible in STRIPS formalism. Consider coloring graph which forms unit cube (Fig. () and whose node embeddings correspond to the latent space of some images. A cube-like graph on the left can be efficiently colored (i.e., by fewer colors) by the difference between the neighboring embeddings. Edges can be categorized into 3 labels (0. 0, (1)), (0, e),0) and (@1, O, 0) (6 labels if directed), where each label is assigned to 4 edges which share the same node embedding differences, as de- picted by the upward arrows with the common node difference (0, O, @1) in the figure. The set of node embedding differences corresponds to the set D, and each element of D represents an action, decoded into images, moving toward positive direction of x--
e.g., when the node embeddings axis may result in the "0" tile in the image moving to the right, and moving toward positive direction of z-axis may result in the "2" tile in the image moving to the right effectively making actions compositional. In contrast, the graph on the right has node embeddings that are randomly shuffled. Despite having the same topology and the same embedding size, this graph lacks the common pat- terns in the embedding differences like we saw on the left, thus cannot be efficiently colored by the node differences.
No duplicated edges hetween the same pair of nodes
15

---

Under review as a conference paper at ICLR 2021
(0.1.1)
(0,1,1)
(1,1,1)
(1,0,0)
(0.0.1)
(1,1,0)
10.0,+1)
(0.1,0)
(0,0,+1)
(1,1,0)
(0,0,0)
(xy.2)=(0,0,0)
(x,0,)
(1,0,0)
(1,0,1)
decode
Figure 8: (Left) A graph representing a 3-dimensional cube which is a cube-like graph. (Right) A graph whose shape is identical to the left, but whose unique node embeddings are randomly shuffled.
As such, cube-like like graphs can characterized by the edge chromatic number (minimum edge col- oring) according to the node differences. We now discuss some theoretical properties of the graph coloring with and without the assumptions on the colors and the node differences.
Theorem (Edge chromatic number Vizing (1965)). Let the edge chromatic number c(G) of an undirected graph G be the number of colors in minimum edge coloring. Then c(G) is either A or A + 1, where A the maximum degree of the nodes.
Theorem 2. Mapping F. D provides edge coloring and thus c(G) S minn |D]
Proof. f is one-to-one; f w/ = f(w) + [ww". For any set X, f(w) o f(w') = X @ f(w) + X @ (ww). For any adjacent edges (v. w) and (e, w'), w + w' because is simple (at most edge between any nodes). thus f(t) @ f f(w) * f(v) f(w"). The remainder follows from the definition.
For Thm. 2, equality holds for hypercubes. For certain embedding dimension |S]. there are graph instances where cGG) < min D (Fig. left). We "proved" this with an Answer Set Programming solver Potassco Gebser et 2011
(:action cl :precondition (and) :effect (and (when (and (z0) (not (zl)) (z2))
4
7 de
2
5
8
b
C
(and (z0) (z1) (not (22))))
3
6
9
f
(1=0010,1) color:cl V,=(1.1.0)
Figure 9: (left): An undirected graph consisting of 5 disconnected 2-star graphs It has cGG) 2. When [S] = 4, no assignments satisfy [D] 2. (An assignment with | = 4 exists.) (right): Single conditional effect can encode an arbitrary transition.
Based on these results, we next consider the minimum number actions required model an undirected cube-like graph G(S, D) as a planning model. We restrict ourselves to a precondition- free planning domains in order focus on the action effects. We first need lemma in order focus on undirected graphs.
Lemma 1. Let P be a precondition-free grounded STRIPS planning problem which contains ir- reversible actions. There is corresponding planning problem P' whose state transition graph is identical to that of P and whose precondition relaxation p" is reversible.
Proof. For any irreversible action a in P, add a set actions A (a) whose size [a=1 = 2) ()nn((a)_nn((()))) Each action E A-)(a) contains those effects which encode one of the possible non-deterministicc outcomes of reversing a, and contains unsatisfiable precondition (e.g., adding a new proposition whose value is constantly false). Then the state transition graphs of P and P' are identical and p" is reversible.
16

---

Under review as a conference paper at ICLR 2021
Since assume precondition-free domains, for any P we could instead consider P" to discuss the effects of reversible planning domains. Let be an undirected cube-like graph which is isomorphic to the state transition graph of peccondition-fre reversible planning model P.
Theorem 3. Let Pe be another planning problem definition which models G. The action effects in Pe are allowed to use conditional effects. Then the minimum number of actions in Po required to model G cG)
Proof. For each color e E I..c(G) and for each edge (v, w) colored c. and for each propositional value f(w);; E (0, 1}. we add a conditional effect to e-th action. If [(w), = 0, add a delete-effect, and f(w); 1, add an add-effect. The effect is conditioned by the full conjunction of f(v) using negative preconditions. See Fig. 9 (right), where all effects are put in one conditional effect.
Theorem 4. The minimum number of actions required to model G without conditional effects (i.e. by STRIPS effects) is miny 2 D).
Proof. Each d E D needs 2 actions for forward/backward directions.
Thm. indicates that conditional effects can compact as many edges as possible into just = A or A actions regardless of the nature of the transitions, while Thm. and Thm indicate that STRIPS effects require a larger number of actions. Therefore, merely assigning action symbols to state transitions (which is equivalent edge coloring) does not result in a compact STRIPS model. Notice that the vanilla MLP AAE binds an unrestricted edge chromatic number c(G) A or A + by the maximum number of action labels A (a hyperparameter of the network), but does not bind 2 DI, the edge chromatic number in terms of neighboring node embedding differences. In order to find compact STRIPS action model, we should instead bind 2 by A and restrict latent state transitions to follow a STRIPS transition rule.
In order to constrain the learned latent state space of the environment to a cube-like graph, we pro- pose Cube-Space AutoEncoder. We first explain a vanilla Space AutoEncoder, an architecture that jointly learns the state and the action model by combining the SAE and the AAE into a single net- work. We then modify the APPLY progression to restrict state transitions. Due to the flexibility of neural networks, the loss enhanced by the restriction automatically propagates to the state represen- tation, i.e., it modifies a state representation in order to reduce the loss produced by the restricted action model.
A.3.3 VANILLA SPACE AUTOENCODER
The vanilla Space AutoEncoder (Fig. 10 right) connects the SAE and AAE subnetworks. The nec- essary modification, apart from connecting them, is the change in loss function. The original AAE was trained to optimize the distance between the binary vectors using binary crossentropy (BCE), which is assymmetric in definition: BCE(x, y) = x log y + (1 x) log(1 y). While this was not problematic in the AAE which uses the fixed state representation and the successor prediction y, it is more natural for Vanilla Space AE to use a symmetric loss that equally affects x and
In addition to the loss for the successor prediction in the latent space, also ensure that the pre- dicted successor state can be decoded back to the correct image a².1 Thus, the total loss a sum of the losses for: (1) the main reconstructions f(o*00 zi.0) and E( g (2) the successor latent state reconstruction f(2i.1 1(,1), (3) the image reconstruction from the predicted successor !(o gf.,1 and (4) the KL regularization. We call the second term as direct loss.
We next formally analyze this training objective. Given observed transition (o", o'), assume that の follows (o oo) and o follows (ā")) 01) (See Sec. 2.2 for explanation). The maximiza- tion objective is log- likelihoodd of observing pair of states oº
We iteratively derive the lower bounds (ELBO) by inserting several latent variables. We first intro- duce 20, zt:
17

---

Under review as a conference paper at ICLR 2021
Space AE
Back-to-Logit (BTL)
z
1
apply
continuous repr.
0
discrete repr.
Normalization
Batch
BN
sum
BC
BinConcrete
Z
MLP
action
apply
1,1
a
effectt
discrete repr.
z
loss
a
Simplified First Order State AE (SimpleFOSAE)
Raw observation
z/1
Ô
decode
@ (1,2)
(1,2)
@ 2200
8
(2,0)
Object detection
@ (0,0)
Z/2
(0,0)
(preprocessing)
(Input) Object feature vectors
pointwise conv
Reconstruction
Figure 10: The illustration of State AutoEncoder, Action AutoEncoder, and the end-to-end combi-
nations of the two.
log peop(,oooo 0²) )² 2 - -Dl x² o (l))ll(z",22') + + p(o of ,²,2''1
(1)
The first term of Eq. 1 is the KL divergence for 20 2 Since all latent variables are assumed to be independent (mean-fie field assumption), p(=", ; zz = >p(=")((('' where p(="),p) p(z are respectively the prior distributions for the binary latent variable n and =1 that discussed in Sec. ??. q(2", 21 o o²)) can also be decomposed because, the encoder modeled by y, = depends only on o and = depends only on o Therefore, the entire KL divergence is divided into individual KL divergence:
DKL(g(2 o0, 1p(2"" 1p((=",z') = =Dxl(q(z' oo o (')llp(())) + DKi(q(z) 0o, o ol)ll((z)))
=Dxi(q(2" I | (lp())) + DRL (g(z" oz) (2)
We next decompose the second term in Eq as shown in Eq This derivation is possible because σ and ot are generated independently the decoder (the network that produces 8 ad from 2", 2²) modeled by p. The first term in Eq corresponds to reconstruction loss for の (MSE due to p(o" 2) NoO". oo)).
log p(pp o' 1=",²') logp(o" =²xx²) )²) + log p(o') 1=0,²')
=logp p =²) + log 2² 2²,²
(3)
Next, we derive the lower bound of the second term in Eq. by introducing a ooe-hot categorical latent variable for action labels, and its prior distribution p(a 2 21) Cat(1/A) (uniform categorical distribution of categories).
log p 1=", ,²,,²²)
DRI.(g(e o² o²,20 = 2²,,²1)) 2².21) _=
+ Eyyaajo'
(log))/lopppeeee a, ,²,²1)]
(4)
Finally, to complete the Vanilla Space AE network, we should model the second term in Eq Jusing another latent variable 31, We further derive the lower bound using the same reformulation:
18

---

Under review as a conference paper at ICLR 2021
aa,2,. poo' a, 2. 2²) (al(a(2 a a² 2,) a z 2²)) E Q(2 of 22, Dog 1 ,",a,, ,2,2)))
(5)
The second term of Eq. is reconstruction loss for o! which can be computed by assuming p(o* 21 a, =², =²) (ō m. However, what should we use as a prior distribution p(ž* a, z", =!) in the first term (KI. divergence)? We set it to be g(= o²), because the choice of a prior is arbitrary and want that the distribution of the predicted successor state x² to be identical to the distribution of the successor state x² directly encoded from the input. As a result, the total maximization objective is derived as follows:
logp(o'.o' DKL(g(>0)
Interpretation
o )p)((()))
KL divergence for 2 in Eq
DKL(9(>)
0² I[p(="))
KL divergence for 2 in Eq
+ +ogg((o" 2²
Reconstruction loss E(O", 0%) in Eq
-DiLl(aa oo 20, 1 = llpla 2², x²,=²))
KL divergence for a in Eq
- -Ki(a(>² o',a,, a,
z z')ll((=' (o))
KL divergence between z' and 2 in Eq direct loss 8(z ,
+ log p(o" /2 "3,,a, a, a, 20, 2²).
Reconstruction loss E(o', 8² in Eq.
(6)
Direct loss ((z), 2 be computed by the same method introduced in (Sec. ??). We assume q(=1 o¹
,',a, a x0 ) Bernoulli(')) and g(*1 o') Bernoulli(q') where q²' a(i),
q' = o(l'). and i 1 are the inputs to the corresponding Binary Concrete activation. Then the KL divergence is computed as
DKL(g(ž' 1o', a, a,. ",,z%,²))llfr )l |o = DKL (Bernoulli( )} [Benoouli(q"))
q'
(1 q log - q
q¹ log q
q²
Coincidentally, this DxL o², 20, aa) (0.5) log 2 BCE(Q' q²),, whose last binary cross entropy term similar to the loss function of AAE.
Implementation note: we instead optimize (g o² , a (Brrooull)(())) + log + BCE(21, 12),, the training becomes slower due to the unnecessary logistic noise inside BC of s = BC((') and z1 = BC(()).
Implementation note: It is important to bootstrap various terms in the optimization objective. In our experience, delaying the application of the direct loss until a certain epoch seems crucial, and suspect the reason is due the relatively short network distance between the two latent spaces 2:.1 /g compared to / / we enable the direct loss from the beginning, the total loss does not converge because it1 prematurely converges to i causing a mode-collapse (e.g. all 0), before the image en/decoder learns a meaningful latent representation.
A.3.4 CUBE-SPACE AE
Cube-Space AE modifies the APPLY network so that it directly predicts the effects without taking the current state as the input, and logically computes the successor state based on the predicted effect and the current state.
naive implementation of such network shown in Fig. (left). The EFFECT network predicts binary tensor of F x 3 using F-way Gumbel-Softmax of categories. Each Gumbel Softmax corresponds to one bit in the F-bit latent space and 3 classes correspond to the add effect, delete effect and NOP, only one of which is selected by the one-hot vector. The effects applied to the
19

---

Under review as a conference paper at ICLR 2021
Naive method
"Back-to-Logit"
i,1
min
max
BN
1,0
zi.
BinConcrete
Dell
Add
Nop (ignored)
BN Batch Normalization
a
Ded
a effect
efffcc
+Nop=1
Figure 11: A naive and a Back-to-Logi implementation of the APPLY module the Space AE.
current state cither by max/min operation its smooth variants (smooth min/max). Formally, the naive Cube-Space AE is formulated as follows:
B zz APPLY( a') min(max( (2" ADD(() (EL()))))
(8)
smin(smax( ADD(()a)), 1 ((Il())))), where
(9)
EFFECT(()a)) GS(MLP((') E BFx (EL(()a)) EFFECT(())*)) smax(x, v) log(e* +
ADD(()a) EFFECT(()a')o NOP(a') EFFECT(()a))
smin( y) -log(e - +e
(10)
While being intuitive, we found these naive implementations extremely difficult to train. Our con- tribution the architecture is Back-to-Logit (BtL. Fig. I1 right), generic approach that computes a logical operation in the continuous logit space. We re-encode a logical, binary vector back to a continuous, logit representation by an monotonic function m. This monotonicity preserves the order between true (1) and false (0) even after transformed into real numbers. We then apply effects by adding the continuous state vector with continuous effect vector. The effect vector produced by applying an MIP named EFFECT to the action vector a'. After adding the continuous vectors, we re-activate the result with discrete activation (Binary Conerete). Formally,
i1l APPLY a²) = (C + EFFECT(()a))
(11)
We found that an easy and successful way to implement m is Batch Normalization loffe & Szegedy (2015), method that was originally developed for addressing the covariate shift the deep neural networks.
For simplicity, we consider scalar operation, which can be applied to vectors element-wise. During the batch training of the neural network, Batch Normalization layer BN(x) takes minibatch input = {x1 ...II }. computes the mean and variance o², then shift and scale each I² so that the result has mean of and variance of 1. It then shifts and scales the results by two trainable coefficients y and B. Formally,
Additional background:
Vx' E B; BN(x')
# 0 +
(12)
While rescaling the normalized result by Y and 3 seem negate the original purpose of normal- ization, the presense of normalization to mean 0// variance is crucial. Notice that the first scaling depends other training examples in the same minibatch, while y and 8 is not dynamically ad- justed for the minibatch. For example, imagine two minibatches B1 and Bg, where B₁ accidentally tends contain larger values than B, but the variance within B, and B, are the same. The bias is canceled by the normalization, and thus the outputs rescaled by y and B are computed based on a relative scale inside the corresponding minibatch.
Implementation note: Since a' is a probability vector over A action ids and a' eventually converges one-hot vector due to Gumbel--oftmax annealing, the additional ML. be merely linear
20

---

Under review as a conference paper at ICLR 2021
embedding, i..., EFFECT(()a") Ea', where E RPx and BF. It also helps the training if we apply BatchNorm on the effect vector. Therefore, a recommended implementation is
APPLY(a, (") = BC(BN(Z**)) BN(EEa))
(13)
where (FFECT(((') BN(EE'))
A.3.5 BACK-TO-LOGIT AND ITS EQUIVALENCE TO STRIPS
States learned by BTL has the following property:
Theorem 5. (Asai & Muise 2020) Under the same action state transitions are bitwise monotonic, deterministic, and restricted to three mutually exclusive modes, fer. each bit j:
(add:) Vi; (2)
(.11
E (1, i.e. +.0 S 1.1
(14)
(del:) Vi; (z) ] E (0,00) ie. 2/ 2 1.1
(15)
]
(nop:) Vi: 4.0 (.1 ((0,0), ie 1.0 1.1
;2
2]
(16)
This theorem guarantees that each action deterministically sets a certain bit on and off in the binary latent space. Therefore, the actions and the transitions satisfy the STRIPS state transition rule = (s (EL()))) U ADD(()a), thus enabling direct translation from neural network weights to PDDL modeling language.
The proof is straightforward from the monotonicity of the BatchNorm and Binary Concrete. Note that we assume BatchNorm's 's additional scale parameter y is kept positive or disabled.
Proof. For readability, we omit j and assumes a I-dimensional case. Let e (FFECT((a)) e I. which is a constant for the fixed input a. At the limit of annealing. Binary Concrete BC becomes STEP function, which is also monotonic. BN is monotonic because we assumed the scale parameter y of positive, and the main feature BN also only scales the variance of the batch, which is always positive. Then we have
11, STEP(BN(Z20) +e..
(17)
The possible values pair (2*+0, ) can have is (0. (0. 1), (1, 0), (1, Since both STEP and BN are deterministic at the testing time (See loffe & Szegedy (2015)), we consider the determin- istic mapping from 2 zi11 There are only deterministic mappings from {0, to {0, 1}: 1), (1, 1)), {(1, 0), (0, 0)}. {(0, 0), (1, 1)}. {00, 1), (1, 0)}. Thus our goal is to show that the last mapping {(0, 1), (1,0)} is impossible in the latent space {... {...(z*) (z 2(1) produced by BTL.
To prove this, first, assume (2.0 2(2) = (0. 1) for some index i. Then
= (T))(BN(() + (N + 0 + BN(1) + 0 = Vi; BN(2
0.
(18)
The second step is due to the monotonicity BN( < BN((). This shows z is constantly regard- less of 20.0 therefore it proves that (z*.0, 211) (1, 0) cannot happen in any i.
Likewise, if (zf00 2111 = 0) for some index i,
0 = ==STEP(BN(1) + BN(1) + < 0 BN(() + < 0 > Vi; BN(2 +e<0 O.
(19)
Therefore, zdd 0 regardless of z and thus (2' (0, 1) cannot happen in any
Finally, if the data points do not contain (0, 1) or (1, then by assumption they do not coexist.
Therefore, the embedding learned by BTL cannot contain (0, 1) and (1,0) at the same time.
A.4 EFFECT RULE EXTRACTION
To extract the effects of action from Cube-Space AE, we compute ADD(A) APPLY(a. 0) and (a = 1 - APPLY (a, 1) for each action a, where 0, are vectors filled by zeros/ones and has the same size the binary embedding. Since APPLY deterministically sets values 0 1, feeding
21

---

Under review as a conference paper at ICLR 2021
these vectors is sufficient which bit it turns and off. For each j-th bit that is 1 in each result, a corresponding proposition is added to the add/delete-effect respectively.
In FOSAE+,, we extracts the effects from parameter- sorameter-bounedd subspace 2/ 1
24,1 The representa-
tion is tuple 2; (2 11...... //..1....///// /N), where 2//n X.xxxxxaxx//n BTL then operates on flattened and concatenated binary vectors of size Σn #a" P/n: The input, the output, and the effect share this shape. We extract the effects from this BTL vector in the same manner as noted above. After the extraction, however, each bit converted to a lifted predicate according to the position. For exam- ple, when a bit that corresponds to 2,/21,2,5 has turned from 0 to 1, then the add-effect contains p5(arr1,, Parg2), where ?arg1 is parameter used in the lifted PDDL encoding.
We show an example of such a learned PDDL model below, obtained from 8-Puzzle with P/1 P/2 = 333 (reformatted for readability). Note that we disabled the nullary predicates P/O and z/0, which consumes the first 333 dimensions in the flattened vector. Another note is that we also count the number of appearances of each action in the training dataset. If an action label is never used in the dataset, it is not exported in the resulting PDDL output. Thus, the index for the action starts from 7 in the example.
(define (domain latent)
(:requirements :strips :legativee-preoondtttons)) (:predicates
(p333 ?x0)
(p665 ?x0)
(p666 ?x0 ?x1)
(p998 ?x0 ?x1))
(:action a7 :parameters (?x0) :precondition
(and (p339 ?x0) (p388 ?x0) (p391 ?x0) (p398 ?x0) (p402 2x0)
(p420 2x0) (p421 ?x0) (p446 ?x0) (p447 2x0) (p473 3x0)
(p475 ?x0) (p489 ?x0) (p491 2x0) (p502 ?x0) (p516 2x0)
(p559 ?x0) (p588 ?x0) (p615 2x0) (p641 Px0) (p648 ?x0)
(p831 ?x0 ?x0). (p950 7x0 ?x0) (not (p333 ?x0))
(not (p371 2x0)) (not (p375 2x0)) (not (p388 ?x0))
(not (p402 ?x0)) (not (p406 ?x0)) (not (p421 ?x0))
(not (p447 2x0)) (not (p454 2x0)) (not (p504 ?x0))
(not (p508 ?x0)) (not (p519 ?x0)) (not (p524 ?x0))
(not (p526 2x0)) (not (p562 2x0)) (not (p584 Px())
(not (p593 ?x0)) (not (p617 ?x0)) (not (p640 ?x0))
(not (p652 ?x0)) (not (p824 ?x0 ?x0)) (not (p892 ?x0 ?x0)
(not (p926 ?x0 ?x0)) (not (p975 ?x0 ?x0))
(not (p994 ?x0 )x0)))
ceffect
(and
(p349 ?x0) (p361
?x0)
(p366
7x0) (p370
7x0)
(p371
7x0)
(p378 Px0) (p381 ?x0)
(p385
7x0)
(p388
7x0)
(p401
7x0)
(p408 ?x0) (p421
?x0) (p432
?x0) (p445
?x0) (p454
2x0)
(p475 ?x0) (p491 ?x0) (p496
2x0) (p502
2x0) (p503 ?x0)
(p504 2x0) (p507 ?x0) (p517 ?x0) (p526 ?x0)
(p550 2x0)
(p562 ?x0) (p563 ?x0) (p575 2x0) (p584 2x0) (p588 2x0)
(p599 ?x0) (p601 ?x0) (p607 3x0) (p612 7x0) (p617 ?x0)
(p631 ?x0) (p640 ?x0) (p641 ?x0) (p647 7x0)
(p656 Px0)
(p663 ?x0) (p724 7x0 7x0) (p768 7x0 ?x0) (p831
?x0 ?x0)
(p902 ?x0 ?x0) (p911 ?x0 ?x0) (p993 ?x0 ?x0) (not (p339 ?x0)) )
(not (p355 ?x0)) (not (p365 ?x0)) (not (p391 ?x0))
(not: (p397 ?x0)) (not (p398 2x0)) (not (p402 Px0))
(not (p406 ?x0))
(not (p422 Px0)) (not (p446 PxC))
(not (p447 ?x0)) (not (p448 ?x0))
(not (p451 ?x0))
(not (p456 ?x0)) (not (p472 ?x0))
(not (p473 ?x0))
(not (p478 ?x0))
(not (p489 ?x0))
(not (p490 ?x0))
(not (p495 2x0))
(not (p516 2x0))
(not (p518 ?x0))
(not (p524 ?x0))
(not (p525 ?x0))
(not (p527 ?x0))
(not (p534 2x0))
(not (p544 ?x0))
(not (p559 Px0))
22

---

Under review as a conference paper at ICLR 2021
(not (p561 )x0)) (not (p615 2x0))
(not (p624 ?x0))
(not (p629 2x0)) (not (p642 ?x0)) (not (p646 ?x0))
(not (p651 2x0)) (not (p653 2x0)) (not (p720 ?x0 ?x0)) )
(not (p813 7x0 ?x0))
(not (p824 7x0 ?x0)) (not (p892 7x0 ?x0)) )
(not (p894 2x0 2x0))
(not (p926 7x0 ?x0)) (not. (p931 7x0 2x0)
(not (p975 ?x0 ?x0))
(not (p994 ?x0 )x0))))
(:action a22 :parameters (?x0) #precondition
23

---

Under review as a conference paper at ICLR 2021
A.5 PRECONDITION LEARNING WITH DYNAMICS REVERSED IN TIME
In the main text, we simplified the model by showing only the forward dynamics, i.e., the dynamics in the same direction as the time. This forward dynamics can model the effects (add/delete) of the actions. However, the forward dynamics is insufficient for learning the preconditions of the actions. The original CSAE paper (Asai & Muise 2020) used an add-hoc method that extracts common bits of the current states.
In contrast, we added network that uses the same BTL mechanism that applied backward in time, ie., predict the current state 2 from successor state 2t1 and a one-hot hot action vector a'. We named the network REGRESS(2 aa)), alluding to the regression planning (Alcázar al. 2013) literature.
In REGRESS, add-effects and delete-effects now correspond to positive preconditions and negative preconditions. A positive precondition (normal precondition) requires that proposition is T prior to using an action. In contrast, negative precondition requires that a proposition is 1 prior to using action. While negative preconditions are (strictly speaking) out of STRIPS formalism, is commonly supported by the modern classical planners that participated in the recent competitions. To extract the preconditions from the network, effects from the progressive/forward dynamics.
the same method used for extracting the
The entire model thus looks follows.
(encoder)
2 z ENCODE(O O',)), ENCODE(O
(action parameters, uses NLMs)
x² PARAMS (2 2.0 2.11)
hooudd subspace extraction)
40
.1 BIND 2 100 BI',,IIN(2"* .xx)
(action assignment)
ACTION (2 ; 22111)
(bounded forward dynamics)
21. APPLY 2t 6.0 ,aa)
(bounded backward dynamics)
2 REGRESS (2) i1l aa)
(reflection to global forward dynamics)
1t. 4.0
UNBIND 1.0 x²) UNBIND(2
(reflection to global backward dynamics)
0 241 UNBIND(2 x²) UNBIND(2
(reconstructions)
0 8 DECODE (')), DECODE(Z*!)
(reconstruction based on forward dynamics)
gi.1 DECODE
(reconstruction based on backward dynamics)
%1.0
DECODE
The total loss is E(o+.) +E(021
ofth + ) 600 )+((21)
3f ) ++(0)),,,)))))) )
( 2000 {(2 21.0 + Reg.
24

---

Under review as a conference paper at ICLR 2021
B IMPLEMENTATION DETAIL
We based our code on a publicl--available Latplan source code repository com/ com/guion cho 1828/ /tt///aplaa////n which is based on Keras Deep Learning library. The reposi- tory hosts its own Genetic Algorithm based hyperparameter tuner which we mention several times later.
B. INPUT DATA FORMAT AND LOSS FUNCTIONS
As we discussed in the earlier appendix section, our final loss function consists of 9 terms: gi.0 f(o 0².0 + ,, " g1.1 )+((0''
) +x)x())))))))((((( + x3 + x0
((2)" 1. =; + Reg. We first describe the input data format that is shared among different domains, then the loss functions defined on it.
Our input/output format OF consists of O objects (enviromment-depeeddenl each having F features, F' features consists of image-based features and the coordinate / dimension-based features (Fig. D2.. All image patches extracted from the observations are resized into a fixed height H, width W and color channel C' 3. Each flattened object vector has the size F = H x W C + 4. The last dimensions contain the center coordinate and the actual height/width before the resizing. Out of the terms in the total loss, the terms that apply to the object vectors of this form are L(o o(.))
((o'', .o:1). E(o':1 zi.1) !(o'-0 g1.0)
central coordinate
image data ( (HxWxC)
h
dimensions
Figure 12: Data array representing a single object.
For this data format, the loss function consists of the mean square value of the image part, and the square sum of the coordinate dimension parts. We do average the losses for the coordi- nates/dimensions to avoid making the gradient minuscule. To further enhance this direction, we additionally have the coordinate loss amplifier À tuned by GA, location has more visual impact on the reconstruction. Note that, for the tuning with the validation set and the evaluation with the test set, we set À measuring. À is altered only during the training. Formally, for the i-th objects in the input o and the reconstruction ö, we define the loss as follows. These losses are averaged over the objects and the batch dimensions.
often the case that the object
order to obtain the consistent, comparable
f(o o,) HWC | o,,.,wellB + Allo,, - o...w..r
We call the remaining losses except the regularization terms as "latent dynamics loss". They oper- ate on the binary latent data activated by BinaryConcrete: f(zt.1 z(2), {(z,,) {(z 25.0). However, note that during the training, all values used for the loss calculation are still continuous due to BinaryConcrete's annealing. This means we can't use Binary Cross En- tropy BCE, the standard loss function for binary classifications, because the "training data" is also noisy probability value. is also fact symmetric - as we discussed in the previous appendix sections, the role of these losses is not only just to obtain the accurate dynamics, but also to shape the state representation toward a cube-like graph. While there are several candidate loss functions that can be considered, signed for noisy labels, which simply applies BCE both ways. Formally, given BCF(2, Σ, (z, log 2 + (1 2,) log(1 2,)).
.x(1) f(210 gi.),,
adapted Symmetric Cross Entropy (Wang et al. 2019) de-
(z, = BCF(2, BCF(2,2).
The remaining regularization losses include KL divergence for Discrete VAEs, as well as the regularization for the latent vectors 240 z1.l which were proven useful in Asai & Kajino (2019)
25

---

Under review as a conference paper at ICLR 2021
Finally, define the magnitude and the warmup of each loss. The magnitude multiplies each loss and is tuned by the GA tuner. Similar to X, the values are set to during the evaluation. We have a for the LI regularization, for KL divergence, and y for the latent dynamics loss.
The warmup mechanism works by setting these values to 0 until the training reaches a certain epoch defined by the ratio relative to the total number of epochs. We used the warmup fa- ryy for a, 7, as well as 'ree and the rdyn for the main reconstruction loss E(o o².0) E(ot.) ...1) and the dynamics-based reconstruction loss {(o ,0,'²) l(o 20,.0
The motivation behind these magnitudes and warmups to balance the speed of convergence of the various parts of the networks. Depending the hyperparameters (depth, width of the layers), occasionally the network completely ignores the dynamics, falling into something similar to (but the mechanism will be very different from) mode collapse where the effect caused by the dynamics is empty, e.g., the forward dynamics produces the same state as the current state.
Another failure mode of the network is that the dynamics loss is too strong, due to its BCE which could become too large compared to the reconstruction loss. As a result, the network learns a perfect but meaningless latent space dynamics which does not produce correct reconstructions, Tuning the warmup and balancing the losses addressed these issues.
B.2 NETWORK DETAIL.
FOSAE++ consists of 6 networks: ENCODE, PARAMS, ACTION, APPLY, REGRESS, DECODE. Note that BIND and UNBIND are weight-less operations.
B.2.1 ENCODER
ENCODE can be divided into trivial continuous feature extraction phase (pre-encoder) and the actual FOSAE encoder. The feature extractor is simple ID pointwise convolution over the objects, i.e., same as applying the same and the hidden layer width is tuned by the GA tuner. All activations except the last Binary Concrete are Rectified Linear Unit (Fukushima) 1980 Nair & Hinton 2010). The architecture illustrated in Fig. 12
layer on each individual object. Its depth
We should note that we assign the same number of predicates to different arities: P/1 P/2... P/N. In this Network Detail section, denote P P/1 ... P/N hyperparameter that specifies the number, although P was already used in the main text as the total number of predicates (i.e., En P/n).
Pre-encoding step with 1D pointwise convolution (continuous feature extraction)
Main encoding step with 1D pointwise convolution
multi-arity predicate representation
Input
convi (oopoutt0.11-commt
convt D-relu- DD-rrlu-BN--DroouttO.. Dropoot(O.1)) -con/I-BBinCoccetee
z/1
object
abjert
abjert
object
enumerate concatenate
Hww.--reluuB--Dopouuue MiwiID-BinConcrtte
2/2
object 1.1 object 1,2
object 4,3
object 4,4
The encoder has two stages. The first stage extracts a compact but still continuous repre- sentation of each object. From this compressed representation, the second stage identifies properties of objects as well as relationships/predictes between objects.
Figure 13:
26

---

Under review as a conference paper at ICLR 2021
B.2.2 PARAMS
PARAMS consists of multiple NLM layers, depicted in Fig. 14 and the width Q of PARAMS is tuned by the GA tuner.
All hidden activations are ReLU,
multi-arity predicate
multi-arity continuous
Continuous
representation
Logit
representation
shape:
OxP OxOxP OxOxOxP
OxQ OxOxQ OxOxOxQ
Oxia OxOx#a OxOxOx#a
objects being attended
#axO
transpose
NLM-relu- bn-dropoutt0.17
2/2
/2
12
z/3
/3
13
arg arg3
#a-way Gumbel-
arg2 arg3
Softmax
Figure 14: PARAMS network using NLMs.
B.2.3 ACTION, APPLY REGRESS
Since ACTION takes parameter-boundel representations (z/") =x(1) whose objects already lected and appropriately reordered by BIND, simply apply MLP whose last layer has output size A, and is activated by Gumbel-Softmax. This results in selecting one action, represented by a one-hot vector. Before applying the MLP, should flatten and concatenate the tuple of vectors, each of size #a x .. x #a X P. The depth and the hidden layer width of the MLP is automatically tuned.
For APPLY and REGRESS, we similarly flatten the input and directly apply the BTL structure de- scribed in the earlier sections. The EFFECT network in gt1 = BC(BN(2*0) EFFECT((")) is a sin- gle linear layer without bias combined with batch normalization. In other words, with the weight matrix W.
+ BC(BN(2*-0) BBN(W'))).
B.2.4 DECODER
The decoder consists of NLMs followed by post-decoder that shares the same width and depth as the pre-encoder. Blocksworld, we additionally made the decoder a Bayesian Neural Network to absorb the uncertainty in the reconstruction. Details available in (Sec. D.2)
B.3 HYPERPARAMETER TUNER
The tuning system assumes that the hyperparameter configuration is vector/dict of categori- cal/finite candidates. The tuner is a textbook integer GA with uniform crossover and point muta- tion. Assume that hyperparameter configuration is represented by It values. A new configura- tion p = (PI.... .PH) created from two parents q = "qu,..., and r = {ri.... ,TH by Vt;p: p; RandomChoice(ge: rz), and then single value is randomly mutated, i.e., for Random ChrioomChhoiee..... (.. Pm +- RandomChoice Valid Vallid/aluesO((p..)) \ {pm)). stores all evaluated configurations and re-evaluates the same configuration.
In the beginning of the tuning process, it bootstraps the initial population with a certain number of configurations. New configurations evaluated by the validation loss, then pushed in sorted list. A certain number of best-performing configurations in the list are considered as a "live" pop- ulation. each iteration, selects parents from the live population by inverse weighted ran- dom sampling of the score, preferring the smaller validation losses but also occasionally selecting second-ties tier parent. Non non-performing configurations will "die" by being pushed away from the top as the algorithm finds better configurations. The evaluation and insertion to the queue is asynchronous and processes can run parallel.
27

---

Under review as a conference paper at ICLR 2021
C TRAINING DETAILS AND HYPERPARAMETERS
We trained multiple network configurations on distributed compute cluster equipped with Tesla K80 and Xeon E5-2600 v4, which is rather old hardware. The list of hyperparameters are shown in Table
In all experiments. we used the total of 5000 state transitions (10000 states) from the training en- vironments. The details of data collection for each domain is available in the later sections. This dataset divided into vaaining/validatiooneesss sets (90%:5%:5%) The Genetic Algorithm hyperpa- rameter tuner uses the validation loss as the evaluation metric.
We set limit of 1500 total runs for each environment, with 100 initial population, and ran maximum 100 processes in parallel for each environment. As an additional trick, to avoid testing unpromising candidates (e.g., those with diverging loss), the epoch parameter is forced to 50 the first 100 configurations and the runs finish quickly. The rest of the runs use these initial populations as the parents, but replaces the epoch with an appropriate value selected from the candidates.
Training parameters
Optimizer
Rectified Adam, RMSProp, Nadam, Adam
Epochs
100, 333, 1000
Batch size
100, 333, 1000
Learning rate
10 10-2 ,...,,.. 10-6
Gradient norm clipping
0.1, 1.0, 10
Initial annealing temperature Trux
10, 5.2
Final annealing temperature Tiin
0.2, 0.5,
Coordinate loss amplifier
1, 10, 100
Network shape parameters
uP
1, 1, 1, 2,
10, 33, 100, 333, 1000
10, 33, 100, 333, 1000
rre-encoderrdecoder output dimension
10, 33, 100, 333, 1000
Encoder hidden layer width
10, 33, 100, 333, 1000
Encoder depth
2
Decoder hidden layer width
10, 33, 100, 333, 1000
Decoder depth
2
Purams/Actiioo/Appy/Reggesss hidden layer width
10, 33, 100, 333, 1000 1, 2,
Params//ectto//Appii//Rgess depth
Params//ccion/Appp//gggess hidden activation
ReLU, tanh
Regularization Parameters
Latent L1 regularization
0.0, 0.010 0.2 0.0, 0.1, 0.3, 0.8,
KL divergence coefficient 3
Latent dynamies loss 7
Warmup parameters
Latent L1 regularization d,
0.0 0.3,
Main reconstruction loss dyec
0.0,0.01,011, 0.3, 0.5, 0.8
Dynamics-based reconstruction loss dayn
0.0, 0.0, 0.3, 0.5, 0.8
Latent dynamics loss d,y
0.0, 0.01, 0.1, 0.2, 0.3, 0.5, 0.8
Early stopping
Explosion detection
10x the loss value at the epoch
Table 3: List of hyperparameters tuned by the Genetic Algorithm.
C.1 REPRODUCIBILITY, TRAINING CURVES ON RUNS
we show 3 training runs each environment with the same hyperparameter configuration found by our tuner. It reproduces an almost identical, stable curves across runs.
In Fig. 15
28

---

Under review as a conference paper at ICLR 2021
truth cube z/3 (boolean tensor) of ternary predicate p(?x.?y.2z)
binding for move(?from, ?to)
where (?from, ?to) = (c,a)
bind(z/3.(c.aa)
a b ?z
?z
product
?from ?to
a
d
C
a
?y
?x
Note: axis ordering has changed.
?x
?y
0
Figure 15: Training curves of 3 runs on the same optimized parameter found by the parameter tuner. The plots are produced by Tensorboard. We can see that the training is quite stable and the curves similar.
29

---

Under review as a conference paper at ICLR 2021
D DOMAIN- WISE DETAILS
D.1 SOKOBAN
GO
Figure 16: Visualized sokoban problems. The first 5 are used for the training, and the used for evaluation. Pink dots depict the goals that the player pushes the blocks onto. Green boxes are already on one of the goals.
We generated 10000 transitions of each training problem using Dijkstra search from the initial state. We shuffled the 50000 transitions, subsampled 5000 transitions out of 50000, then stored them in a single archive. The rendering and other data are obtained from PDDLGym library Silver & Chitnis 2020). Each tile is resized into 16x16 pixels, and the tile ordering is also shuffled.
In order to make the training data dimension consistent and make it convenient for GPU-based train- ing, we performed a so-called Random Object Masking which removes a certain number of objects from each state via random selection. The idea is similar to masking-based image augmentation commonly used in the computer vision literature, but the purpose is different from those: Ours has more emphasis on having the consistent number of objects in each state. For example, Sokoban training problem (leftmost in Fig. (6) has 49 tiles, while the problem I (the second picture) has 72 tiles. In the combined archive, the number of objects is set the smallest among the problems.
We also removed certain tiles that cannot be reached by the player. For example, in problem O, the three floors on the top left corner cannot be reached by the player. Similarly in problem 1, the bottom right corner is not reachable by the player. We performed a simple custom reachability analysis using the meta-information obtained from PDDLGym library. This helped reducing the dataset size and thus the training.
Finally, during the dataset merging, we accounted for the potential location bias caused by the map size difference. For example, if we preserved the original æ, y locations, the tiles tend to be biased around 0, 0 and the location around (æ,, y) = (12, 12) (by tiles) is never occupied by any tile. To ad- dress the issue, for each state pair, we relocated the entire environment by value selected uniformly randomly from a certain width. The width is decided from the maximum dimension of all training problems. i..., 12x12. For example, state pair in problem 4 (which has 10x9 map dimension) will be shifted by random value between 0.. - 10) in w-axis, and 0.. - 9) in y-axis.
D.2 BLOCKSWORLD
We generated 5000 random state transitions using Battorraaistic-Blockwwee dataset Asai (018)) which in turn is based on CLEVR Johnson et al. (2017) dataset generator. It uses Blender 3D ren- dering engine to produce realistic effects such metallic reflection, surface materials and shadows. The objects are extracted from the information available from the generator metadata. We cropped the image region suggested by the metadata, resized them into 32x32x3 image patches and stored them in an archive. The size of the objects reported by the metadata may vary and is noisy due to the camera jitter, object location, and the heuristics used for extracting the objects. This a case even for the objects that are not moved by the random actions.
Each transition is generated by sampling the current state and then randomly moving block. To sample state, we first generate a set of block configurations (color, material, shape, size), then placing them randomly on a straight line in the 3D environment without collisions. When we move block, select set of blocks of which nothing else is on top, choose horizontal location and place it at the lowest non-colliding height. We ensure that the block is always moved, i.e., not stacked on top of the same block, and not on the floor if it is already on the floor.
randomly, pick new
In Blocksworld, we noticed that a same conceptual symbolic action (such as move) may have a non- deterministic outcome in the image space while each individual concept is discrete and deterministic.
30

---

Under review as a conference paper at ICLR 2021
For example, move may relocate block onto floor, but the resulting position is chosen randomly during the dataset generation, i.c., it can be placed anywhere on the floor.
To model this uncertainty in our framework, decoder for Blocksworld domain. The final output o E ROx is produced by two NLMs of output feature size F, each producing the mean and the variance vectors J4, c R and the output reconstruction is generated by the random sampling: = + a cc where c is a random noise vector following normal distribution (0,, 1). During the testing (e.g., visualization), the random sampling disabled and we use the value for rendering.
used Bayesian Neural Network layers in the
D.3 8-PUZZLE
The 8-puzzle domain generator directly included in the Latplan code base. uses MNIST dataset for its tiles and the tile size is set 16. Similarly, generated 5000 random state transitions using the generator.
E PDDL GENERATION AND PLANNING EXPERIMENTS
E.1
EXAMPLE OUTPUT: 8-PUZZLE
O 5
2 5
2
2
0 2
0 4 7 8 O
2 4
1
3 24
3 0 4
3 340
3 3
3 4
3
6 7 8
6 7 g
6 7 0
6 7 8
6 7 8
04
4 O
142 2
4 2
O a
1 4
2 5
3 5 2
352 2
350
3 5
3 4
3
6 7
6
7
6 7 -
7
6
7
7
3 0 1
3
1 O
3 2
3
1 2
3 2
1 2 3 4 4 7 8
O
4 5 2
4
5 2
450
4
105
O 44
6 7 8
6 7 8
6 7 2
7 8
6 7 8
3 2
3 1 2
3 2
3
1 2 2
3 2
O
2 5 7 8 1
4 5 8
4 4 5 8
4 450
4
005
O 045
3
4
6 0 7
6 7 á
7 2
7 8
6 7 7 8
6
3 O 2
3
2
3 2
3
1 2
3
2
O
2 4 7
6 5
6 O 5
6 5
6
4
O 4 5 7
3
5
7 4 8
7 4 8
7 O 8
O 7
6
3 2
3 2
3
2
3 2
3 2
2 3 4 45 6 7
0 1
6 4 40
6 4 5
6 4 5
6 45
O 45
7 1 5
7 8 0
7 0 8
0 7
6 7
31

---

Under review as a conference paper at ICLR 2021
2 5
25
1 2
2
0 2 3 4
0 1 4 7
2 4 8
3 74
3 04
340
3 4
3
6 0 8
6 7 8
6 7 0
6 7
6 7 8
6
4 2 3 5
1
4 2 358 8
1 4 2 350
4 4
O a
O
1 3 4 7 8
2 4 5
3 0
3 343 4
6 0 7
6
7 0
6 7 8
6 7
6 7 8
3 2
3 1 2
3 2
3
1 2
3 2
0 1 2 4 7 1 O
0 5
6 5
6 4
6
4
O 045
3
7 4 748 8
7 4 8
7 O g
O 7 g
6 7 8
6
3 432 2
O 3 2
3 O 2 5 7 4
3
1 2 0 8 4 6 5 2 7 g
3 2
1 4 7
2 5 g
0 5
4
5 8
4
4
O 4
3
6 7
8
6
7
6
7
6 7 8
6
340 0 5 2 2 2
44
3 4 5 0 2 8 0
0 4 3 6 3 2
04
4 2 5
14 2 3 005
1
0 2 3 34 7
0 2 4
0 2
5 2 7 g
5 3
3
6 7
6 7
6 3
6 3 2
7 8
3 6 0 7 g 5
2
3 2 4 7 8 5
2
2 4 4 5
2
3 2 7
O 2
1
6
6 4
6
0
4
6
4 4
4
7 X 5
7 8
6
O 7 8
5 0 7 2 6
2 5 7 O 4 3 6 2
2
2 5 3 7 L O 6 Z
2. >
2
2 6 3 4 7 8
O 2 3 7
O
2
2.
4
3 0 4 7 8
4 7
4
2 5 4 g 7 6
2 4 3
2 5 4 3 6 7
2 O 4 7
2. 5 3 4 0 O 6 7
2
2. 5 3 4 2 7 6
2.
2 6 3 4 5 7 2
O 2 3 5
O 2
O 2
4 7
4
6
7
3 2 5 7
3 3.3 5
3 2 6 7 5 4 6 &
3 O 2
3 A
3 2
3 A 4
3 2 4 6
3 2 4 6
3 A
O 3
? O 4
7 0 5 4 6 R
7 5 0
0 5 7 &
0 4 5
3 4
4 6
4 6
4
6
4 6
6 E
0
6
0 1 7 2 3 6 Z
1544 4
1 5 4 3 7 2 O 6 Y
5 4
1 5 4 3 0 2 6 7 R
04
4 O 5 2 7 R
1 4 2 3 5 0
1 3
4 A 0 5 7 8
0 3
0 2
2
2
5 3
3
3 141 5
3 4
3
6 0
6
6 3 2
6
6 3 2
6
3 4 6
3 1
3 34
3 0
3 0
3 2
3 2
3 2
O 2
8 2 7 S
6
2
6 0 2
6 4 2 3
6
4 2
6 4 0
6 4 5 6
6 4 5 0
6 1 5 0 1 2 O 3 4 5 7
0 4 5
3
4
5
A 0 5 4
3
7 2
3
7 5 3
7 7
7 8
6
6
1 6
2 3 8
2 5
2 5
2 5
2 5 6 3 4 0 7 Z
2 5
I 2 5 3 04 6 7 2
2 5
O 2
O 2 5
6 3 O 4
6 3 4 7
3 4 O
0 3 4 6
3 0 6
3 3 5
3 4
7
6 4 2 5 1 O 3 7 :
7
6
6
6
4 a 1 0
5
4 2
4 A
C 3 0 1 3 5 6 7 Z 3 C 6 4 2 7 8 5
4 O 3 5
0 2 1 3 5 6 3 2
1 4 2 0 3 2
1 4 2 3 5
O 3
O
7 6
3 :
7 3 0
3 0
3 349 5
3 4
5
6
6 2 8
6 7 7
6
7
3 3- 1
34
3
C
2
3 2
3 3
O A
6 8 2
6 0 2
4 2
4 O 7
6 4 5 7 R 6
4 5 0 8
6 4 5 0 7 &
O 4 5 7 7
3 4 5 7
0 7 5
7 O
7 5 5
7 5
32


Reviews:
[{"review_id": "ZXpN5AGj_bq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of learning dynamics model directly from raw sensory inputs. The authors propose an unsupervised end-to-end model that can perform high-level tasks planning on raw observations. This work extends Asai et al. 2020, 2019 etc, and with improved symbol generation and lifted PDDL. The authors follow the experimental setup as seen in prior work, where three artificial environments (blocksworld, MNIST 8-puzzle, and sokoban) are used for planning. \n\nPros.\n+ interesting and important problem of high-level tasks planning with potentially practical usage\n+ detailed analysis of experiments in the supplementary\n+ the model seems to be working really well without much supervision\n+ training details are provided for re-implementation\n\nCons. \n- The notations used throughout the paper can be a little bit confusing. Readers who have not read supplementary and prior works may find it difficult to understand. \n- Experiments are limited to relatively simple environments. It would be interesting to see how the proposed model can be applied to the more realistic scenarios as seen in Konidaris et al. \n\nOverall the reviewer thinks this paper addresses an important yet less studied topic, and will likely inspire follow-up works towards this direction. While the paper has some problems in terms of presentation, the reviewer recommends its acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IPIoGgCo6Fv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents FOSAE++, an end-to-end system capable of producing \"lifted\" action models provided only bounding box annotations of image pairs before and after an unknown action is executed. Building on recent work in the space, the primary contribution of this work is to generate PDDL action rules. To accomplish this, the authors introduce novel 'params' function that use the Gumbel-Softmax function to implement a differentiable mechanism for selecting which entities are relevant to the current action and feeds them into the new 'bind' and 'unbind' functions that select those elements in the tensor predicting their relevance. Overall, this work is a meaningful contribution in the direction of generated lifted action models without direct labeled data.\n\nThe biggest flaw in the paper is the rather sparse results section. Additionally, I do somewhat take issue with the statement in 4.2 that \"due to the time constraint\" planning experiments beyond the 8-Puzzle domain. Though I appreciate the author's candor in this regard, at the moment the paper is rather weakened by the lack of inclusion of additional planning experiments, particularly since the 8-Puzzle domain is arguably the easiest domain from which one might generated a lifted action representation, due to the black background surrounding the digits. Seeing planning experiments in the Sokoban domain would greatly strengthen the paper. The reconstructed Sokoban environments in Figure 4 look rather good, though the slightly shifted tiles in the reconstructed scene results in black lines/gaps between the cells, raising questions about the ability of the system to perform planning in these domains.\n\nRelatedly, though I appreciate that the authors are space-constrained at the moment, the results section (and in particular the planning section) is quite short and lacking in detail. More detail, including discussion of the limitations of this approach, would strengthen the paper.\n\nThe Appendix is incredibly thorough and a welcome addition to the paper. It provides helpful additional content that, while not necessary for understanding the paper, aid in understanding and implementation.\n\nSmaller comments:\n- The caption for Fig. 2 should be extended or more annotations should be added to the figure itself. Right now, it is only clear from the body text how these components are used or where these models were introduced in other papers. This change would help clarity.\n- A passage addressing the limitations of the proposed system would be a welcome addition as well. In particular, there seems to be a general assumption that only the regions within the provided bounding boxes will change after an action is executed, something this is not generally true (nor is true in the Photo-realistic Blocksworld domain, in which shadows can change outside the bounding boxes).\n- The caption in Fig. 3 is (I think) incorrect: the final note should read that `move` was simplified to `?from, ?to`.\n- Fig. 12 in the appendix is a duplicate of another figure earlier in the paper. It should be replaced with the _actual_ figure before publication.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "46YDsi-KYm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n \nThis paper builds on Latplan (Asai & Fukunaga 2018) and proposes FOSAE++, a lifted action model expressed in First-Order Logic (FOL), which differs from object-centric representation by being generalized over objects and environments. According to the authors, FOSAE++ is the first system that : \n* uses a white-box action model which is trivially convertible to STRIPS formalism, \n* is invariant to the number/order of objects, \n* uses unsupervised generation of multi-arity predicate symbols.\n\nFOSAE++ is demonstrated on three environments: \n* Photo-realistic Blocksworld\n* MNIST 8-puzzle\n* Sokoban\nIn each environment the input is the set objects obtained using a domain-specific segmentation of the image.\n\n## Reason for score\n\nI'm sorry but by expertise in the domain of this paper is way too limited to be able to do a fair assessment. From the introduction I was not able to clearly understand the contributions of the paper. This is not necessarily due to the introduction, but rather to my lack of expertise in the domain. Many of the concepts are alien to me (PDDL, classical planning, lifted action models, object-centric representations, STRIPS). \n\nThat being said, I did find the introduction particularly opaque and it was difficult for me to clearly identify the contributions of the paper. I was not able to make sense of Table 1 and believe the authors could have done a better job of introducing the different aspects that distinguish their methods from previous work.\n\nI did give a good read to the rest of the paper but, again, am unable to comment deeply on the content. I did find the figures particularly dense and difficult to parse. I was also surprised that more than 2 pages were dedicated to background and previous work. Despite that long exposition, I would have to read a lot of the referenced work in order to understand these preliminaries. Again, this may be due to my lack of expertise however I can't help but think that the authors could have done a better job at bringing me up to speed.\n\nAlso, I'm a bit surprised that the authors claim to have achieved \"unsupervised end-to-end neural system that generates a\ncompact discrete state transition model (dynamics / action model) from raw visual observations.\" given that they rely on domain-specific segmentation code to extract objects, and that their technique only operate on these objects. They claim that \"in principle [the domain specific code] can be replaced by the output of object-detectors such as YOLO\" but I find this unconvincing.\n\n## Conclusion\n\nGiven my very limited understanding, I would only consider my rating as an educated guess based on a small number of details I was able to evaluate.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper described a system for deriving PDDL (Planning Domain Description Language) operator descriptions from unlabeled visual image pairs.  The goal is to construct STRIPS-like descriptions with preconditions, add lists and delete lists for operators that can explain the state transitions seem in the image pairs.  This is combined with a neural form of inductive logic programming (ILP) which historically performs a similar task from logical descriptions rather than images.

While this topic is appropriate for ICLR, the work is fairly incremental and the experiments are limited to the 8-puzzle which, according to a reviewer, is the easiest of the tasks.  In spite of boarder line scores, no rebuttal was provided.  So my recommendation is to not accept the paper.
Decision: Reject