GROUNDING PLANNABLE LIFTED ACTION MODELS
Year: 2020

Abstract:
We propose FOSAE++, an unsupervised end-to-end neural system that generates a compact discrete state transition model (dynamics / action model) from raw visual observations. Our representation can be exported to Planning Domain Description Language (PDDL), allowing symbolic state-of-the-art classical planners to perform high-level task planning on raw observations. FOSAE++ expresses states and actions in First-Order Logic (FOL), a superset of so-called object-centric representation. It is the first unsupervised neural system that fully supports FOL in PDDL action modeling, while existing systems are limited to continuous, propositional, or property-based representations, and/or require manually labeled input.

heading: 1 INTRODUCTION
text: Learning a high-level symbolic transition model of an environment from raw input (e.g., images) is a major challenge in the integration of connectionism and symbolism. Doing so without manually defined symbols is particularly difficult as it requires solving both the Symbol Grounding (Harnad, 1990; Taddeo & Floridi, 2005; Steels, 2008) and the Action Model Learning/Acquisition problem.
Recently, seminal work by Asai & Fukunaga (2018, Latplan) that learns discrete planning models from images has opened the door to applying symbolic Classical Planning systems to a wide variety of raw, noisy data. Latplan uses discrete variational autoencoders to generate propositional latent states and its dynamics (action model) directly from images. Unlike existing work, which requires several machine learning pipelines (SVM/decision trees) and labeled inputs (e.g., a sequence of high-level options) (Konidaris et al., 2014), Latplan is an end-to-end unsupervised neural network that requires no manually labeled inputs. Numerous extensions and enhancements have been proposed: Causal InfoGAN (Kurutach et al., 2018) instead uses GAN framework to obtain propositional representations. Latplan’s representation was shown to be compatible with symbolic Goal Recognition (Amado et al., 2018). First-Order State AutoEncoder (Asai, 2019, FOSAE) extends Latplan to generate predicate symbols. Cube-Space AutoEncoder (Asai & Muise, 2020, CSAE) regularized the latent space to a particular form which directly exports to a learned propositional PDDL model (Fikes et al., 1972). Discrete Sequential Application of Words (DSAW) learns a plannable propositional word embedding from a natural language corpus (Asai & Tang, 2020).
In this paper, we obtain a lifted action model expressed in First-Order Logic (FOL), which is a superset of object-centric (property-based) representation that Machine Learning community recently began to pay attention to1, but has long been the central focus of the broader AI community. In propositional action models, the environment representation is a fixed-sized binary array and does not transfer to a different or a dynamically changing environment with a varying number of objects. In contrast, lifted FOL representations are generalized over objects and environments, as we demonstrate in Blocksworld with different number of blocks, or Sokoban with different map sizes. We propose Lifted First-Order Space AutoEncoder (FOSAE++) neuro-symbolic architecture, which learns a lifted PDDL action model by integrating and extending the FOSAE, the CSAE and the Neural Logic Machine (Dong et al., 2019, NLM) architectures.
The overall task of our system is illustrated in Fig. 1. The system takes a transition dataset containing a set of pairs of raw observations which are single time step away. Each observation consists of multiple visual segmentations of the objects. It learns a lifted action model of the environment
1e.g., ICML workshop on Object-Oriented Learning https://oolworkshop.github.io/
by generating the symbols and emits a PDDL (Haslum et al., 2019) encoding for state-of-the-art planning systems.
Contribution Table 1 contains a taxonomy of existing model acquisition systems in chronological order. FOSAE++ is the first system that satisfies all features readily available in symbolic action model acquisition systems, while not relying on human-derived symbols. FOSAE++ generates unnamed symbols by itself — Effectively addressing the long-standing Knowledge Acquisition bottleneck (Cullen & Bryman, 1988) and the Symbol Grounding problem, showing a future direction for high-level symbolic autonomy.

heading: 2 PRELIMINARIES AND BACKGROUND
text: We denote a multi-dimensional array (tensor) in bold and its elements with a subscript (e.g., x ∈ RN×M , x2 ∈ RM ), an integer range n ≤ i ≤ m by n..m, a concatenation of tensors a and b in the last axis by a; b, and the i-th data point of a dataset by a superscript i which we may omit for clarity. We use the same symbol for a set and its size (e.g., S, and not |S|) to avoid the clutter. Finally, B = [0, 1] ⊂ R. We assume background knowledge of discrete VAEs with continuous relaxations (included in the appendix Sec. A.1), such as Gumbel-Softmax (GS) and Binary-Concrete (BC) (Jang et al., 2017; Maddison et al., 2017). Their activations are denoted as GS and BC, respectively.

heading: 2.1 LIFTED STRIPS/PDDL PLANNING
text: Planning Domain Description Language (PDDL) is a modeling language for a Lifted STRIPS planning formalism (Fikes et al., 1972) and its extensions (Haslum et al., 2019). Let F(T ) be
1(Yang et al., 2007; Cresswell et al., 2013; Aineto et al., 2018; Zhuo et al., 2019; Cresswell & Gregory, 2011; Mourão et al., 2012; Zhuo & Kambhampati, 2013)
2Konidaris et al. (2014) requires sequences of high-level options to learn from, such as [move,move, interact, · · · ] in Playroom domain. Causal InfoGAN cannot deterministically enumerate all successors (a requirement for search completeness) due to the lack of action symbols and should sample the successors. James et al. (2020b)’s PDDL output is limited to unary predicates / properties of objects, thus cannot model the interactions between objects. Also, it requires sequences of high-level options such as [WalkToItem,AttachBlock,WalkToItem, · · · ] in the Minecraft domain.
a formula consisting of logical operations {∧,¬} and a set of terms T . For example, when T = {have(I, food), full(I)}, then have(I, food) ∧ ¬full(I) ∈ F(T ). We denote a lifted STRIPS planning problem as a 5-tuple 〈O,P,A, I,G〉. O is a set of objects (3 food), P is a set of predicates (3 full(x)), and A is a set of lifted actions (3 eat). Each predicate p ∈ P has an arity #p ≥ 0. Predicates are instantiated/grounded into propositions P (O) = ⋃ p∈P ( {p} ×O × #p. . .×O ) , such as have(I, food). A state s ⊆ P (O) represents truth assignments to the propositions, e.g., s = {have(I, food)} represents have(I, food) = >. We can also represent it as a bitvector of size ∑ pO #p.
Each lifted action a(X) ∈ A has an arity #a and parameters X = (x1, · · · , x#a), such as eat(x1, x2). Lifted actions are instantiated into ground actions A(O) =⋃ a∈A ( {a} ×O × #a. . .×O ) , such as eat(I, food). a(X) is a 3-tuple 〈PRE(a), ADD(a), DEL(a)〉, where PRE(a), ADD(a), DEL(a) ∈ F(P (X)) are preconditions, add-effects, and delete-effects: e.g., eat(x1, x2) = 〈{have(x1, x2)}, {full(x1)}, {have(x1, x2)}〉. The semantics of these three elements are as follows: A ground action a† ∈ A(O) is applicable when a state s satisfies PRE ( a† ) , i.e., PRE ( a† ) ⊆ s, and applying an action a† to s yields a new successor state
a†(s) = (s \ DEL ( a† ) ) ∪ ADD ( a† ) , e.g., eat(I, food) = “I can eat a food when I have one, and if I eat one I am full but the food is gone.” Finally, I,G ⊆ P (O) are the initial state and a goal condition, respectively. The task of classical planning is to find a plan (a†1, · · · , a†n) which satisfies a†n ◦ · · · ◦ a † 1(I) ⊆ G.

heading: 2.2 NEURAL PROPOSITIONAL/ACTION SYMBOL GENERATION WITH LATPLAN
text: Latplan is a framework for domain-independent image-based classical planning (Asai & Fukunaga, 2018). It learns a propositional state representation and transition rules entirely from image-based observations of the environment with discrete VAEs and solves the problem using a classical planner. Latplan is trained on a transition input Tr: a set of pairs of raw data randomly sampled from the environment. The i-th transition (oi,0,oi,1) ∈ Tr is a pair of observations made before and after an unknown high-level action is performed. Once trained, Latplan can process a planning input (oI ,oG), a pair of raw images corresponding to an initial and goal state of the environment. The output of Latplan is a data sequence representing the plan execution (oI , . . .oG) that reaches oG from oI . While the original paper used an image-based implementation, conceptually any form of temporal data is viable for this methodology, e.g., an NLP corpus (Asai & Tang, 2020).
The latest Latplan Asai & Muise (2020) has a training phase and a planning phase. In the training phase, it trains an end-to-end neural network called Cube-Space AutoEncoder (CSAE) on Tr (Fig. 2, top left). CSAE is a variational autoencoder modeled by binary and categorical random variables each representing the propositional states and the actions in the classical planning. The dynamics modeled by these actions directly compiles into a PDDL model. The network combines BinaryConcrete VAE to produce binary state representation, and a Gumbel-Softmax VAE to produce a
categorical bottleneck layer which assigns a categorical label to each input. Let o0 and o1 be a pair of observed states in a transition, z0 and z1 be the corresponding binary latent states, and a be the one-hot vector that represents a discrete action label assigned to the transition. CSAE is a variational autoencoder network that can be formalized as follows:
(encoder) z0, z1 = ENCODE(o0), ENCODE(o1) ∈ BF
(action assignment/clustering) a = ACTION(z0, z1) ∈ BA
(learning the dynamics) z∼1 = APPLY(z0,a) ∈ BF
(decoder) o∼0,o∼1,o∼ 1 = DECODE(z0), DECODE(z1), DECODE(z∼1)
where ENCODE, DECODE, ACTION, APPLY are arbitrary multilayer perceptrons. The outputs of ENCODE and APPLY are activated by Binary Concerete, and the output of ACTION is activated by Gumbel-Softmax of A categories (hyperparameter). Assuming a certain set of prior distributions, a lower bound (ELBO) of the log likelihood of observing a pair of states (o0,o1) can be as derived follows (Appendix Sec. A.3.3):
log p(o0,o1) ≥ −DKL(q(a|o1, z0, z1)||p(a|z0, z1))−DKL(q(z∼1|o1,a, z0, z1)||q(z1|o1)) −DKL(q(z0|o0)||p(z0))−DKL(q(z1|o1)||p(z1)) + log p(o0|z0) + log p(o1|z∼1,a, z0, z1)
After the training, it generates a propositional classical planning problem (zI , zG) from a planning input (oI ,oG) and export it into PDDL files together with the action model obtained in (2), which are then solved by Fast Downward (Helmert, 2006), an optimized C++-based solver independent from the neural network. Finally, it obtains a step-by-step, human-comprehensible visualization of the plan execution by decoding the intermediate states of the plan into images.
The A-dimensional one-hot categorical variable ai in the network performs a clustering on the state transitions with the maximum number of clusters A specified as a hyperparameter. The cluster ID is used as its action symbol. The clustering is performed by its encoder, ACTION(zi,0, zi,1) = ai, which takes a propositional state pair (zi,0, zi,1) and returns a one-hot vector ai of A categories using Gumbel-Softmax. AAE’s decoder APPLY takes the current state zi,0 and the action ai and reconstructs a successor state z∼i,1 ≈ zi,1, acting as a progression function APPLY(ai, zi,0) = z∼i,1. APPLY is typically just called a “model” in model-based RL literature.
While APPLY can be any network from the training standpoint, such a neural black-box function does not directly translates to a STRIPS action model, preventing efficient search with state-ofthe-art classical planner. Cube-Space AE (Asai & Muise, 2020) addresses this issue by Back-ToLogit technique (BTL Fig. 2, bottom-left) which modifies APPLY. Latent state transitions learned by BTL guarantees that the actions and the transitions satisfy the STRIPS state transition rule s′ = (s \ DEL(a)) ∪ ADD(a), thus enabling a direct translation from neural network weights to PDDL modeling language. Details of the network, the translation method and the proof can be found in the appendix Sec. A.3.

heading: 2.3 PREDICATE SYMBOL GENERATION WITH SIMPLE FIRST-ORDER STATE AUTOENCODER
text: First-Order State AutoEncoder (Asai, 2019, FOSAE, Fig. 2, bottom) is an autoencoder that takes a set of object feature vectors, identifies its latent predicate representation, then outputs a reconstruction of the input. Unlike prior work on relational modeling (Santoro et al., 2017; Zambaldi et al., 2019; Battaglia et al., 2018), this system obtains discrete logical predicates compatible with symbolic systems. The input is similar to the setting of Ugur & Piater (2015) and James et al. (2020b): Each feature vector can represent each object in the environment in an arbitrary complex manner. In this paper, we use segmented pixels combined with bounding box information (x, y and width, height). Let on ∈ RF be an F -dimensional feature vector representing each object and o = (o1, . . .oO) ∈ RO×F be the input matrix representing the set of O objects. FOSAE generates a multi-arity latent representation. Assume we represent the input with a combination of predicates of different arities. We denote a set of predicates of arity n as P/n (Prolog notation) and its propositions as P/n(O). We denote the binary tensor representation of P/n(O) as z/n ∈ BO× n...×O×P/n, and the latent space is a tuple z = (z/1, . . . ,z/N), where N is the largest arity. The total size of its latent space is ∑n=N n=1 O nP/n. To convert the input o into unary
predicates P/1, we apply a 1D pointwise convolutional filter f1 of P/1 output features over the objects and activate it by BinaryConcrete for discretization, i.e., z/1i = BC(f1(oi)) ∈ BP/1. Similarly, for binary relationships P/2, we apply a filter f2 over concatenated pairs of objects, i.e., z/2ij = BC(f2(oi;oj)) ∈ BP/2. We can extend this framework to an arbitrary arity n. The multiarity latent tensors (z/1, . . . ,z/N) can be flattened, concatenated, then decoded to the reconstruction õ ∈ RO×F by an arbitrary decoder network (MLP in the original paper). There is a potential exponential blowup due to ON parameter combinations. This problem is addressed by attentions in the original work (Asai, 2019), but we omit this feature in this paper for simplicity and due to relatively small N (< 4). We call our variant as SimpleFOSAE.

heading: 2.4 NEURAL LOGIC MACHINE (NLM)
text: The NLM (Dong et al., 2019) is a neural Inductive Logic Programming (ILP) (Muggleton, 1991) system whose inputs and outputs are hand-crafted binary tensors representing propositional groundings of FOL statements. These binary tensors are in the multi-arity representation (z/1, . . . ,z/N) identical to the latent space of SimpleFOSAE. NLM provides a key feature that is necessary for learning a lifted First-Order Logic action model: Invariance to the number and the order of objects in the multi-arity representation. The latter implies permutation equivariance, i.e., when the first n axes of each input z/n ∈ BO× n...×O×P/n are reordered by a permutation π, the output axes are also reordered by π. A more detailed explanation of NLM can be found in the appendix Sec. A.2.
An NLM contains N COMPOSE operations which are applied to the corresponding elements of (z/1, · · · , z/N). Each COMPOSE is denoted as COMPOSEQ,σ(z, n) ∈ BO
n...O×Q, where Q is a constant that specifies the number of output features, and σ is a nonlinearlity. An NLM output is denoted as NLMQ,σ(z) = (COMPOSEQ,σ(z, 1), · · · , COMPOSEQ,σ(z, N)).

heading: 3 LIFTING THE ACTION MODEL
text: In order to obtain a lifted action model in an unsupervised setting, there are three requirements: (1) white-box action model which is trivially convertible to STRIPS formalism, (2) invariance to the number/order of objects, (3) unsupervised generation of multi-arity predicate symbols. To our knowledge, no existing systems fulfill all requirements: (Simple)FOSAE lacks 1 and 2, CSAE lacks 2 and 3, and NLM lacks 1 and 3 (designed for hand-crafted symbolic data).
(FOSAE encoder) zi,0, zi,1 = ENCODE(oi,0), ENCODE(oi,1)
(action parameters selector in NLM) xi = PARAMS(zi,0, zi,1)
(extract the parameter-bound subspace) zi,0† , z i,1 † = BIND(z i,0,xi), BIND(zi,1,xi)
(unsupervised action assignment) ai = ACTION(zi,0† , z i,1 † )
(learning the bounded dynamics) z∼i,1† = APPLY(z i,0 † ,a i)
(reflection to the global dynamics) z∼i,1 = zi,0 − UNBIND(zi,0† ,x i) + UNBIND(z∼i,1† ,x i)
(decoder in NLM) o∼i,0,o∼i,1,o∼ i,1 = DECODE(zi,0), DECODE(zi,1), DECODE(z∼i,1)
Loss: `(oi,0,o∼i,0) + `(oi,1,o∼i,1) +`(oi,1,o∼i,1)+`(zi,1, z∼i,1) + `(zi,1† , z ∼i,1 † ) + regularization.
We now propose FOSAE++ whose overview is shown above, which addresses all issues at once. Its overall design follows the CSAE, which ENCODE s the input, identifies the action ai, APPLY-es the action, then DECODE s the results. It uses FOSAE’s encoder to generate multi-arity latent representation, which is then consumed by NLMs used as a basic building block of other components. This architecture’s key element is a new component PARAMS and a unique pair of operations called BIND and UNBIND, which intuitively reflects the structure of lifted action models. Suppose we model a lifted action (move ?block ?from ?to) in Blocksworld (Fig. 1), with effects such as (on ?block ?to). Since a lifted model always refers to the objects through its parameters such as ?to, it cannot affect
the objects that does not appear in the parameter list, e.g., action (move a b c) cannot affect (on d c) because d 6∈ {a, b, c}. We represent this restriction as differentiable matrix operations. To implement this idea, we first added a new attention network x = PARAMS(zi,0; zi,1) = (x1, . . . ,x#a) which attends to the objects, essentially learning the parameters of the action. We assume all actions have the same number of parameters, and thus #a is a hyperparameter. Each parameter xi is a one-hot hard attention vector in BO ( ∑j=O j=1 xij = 1) activated by Gumbel-Softmax. This can later extract a specific object from the object feature matrix o by an inner product xi · o (BORO×F → RF ). PARAMS has several NLM layers, ending with NLM#a,IDENTITY. We then extract its unary part of the results (→ RO×#a), transpose it (→ R#a×O), then apply #a-way GumbelSoftmax of O categories (→ B#a×O). As a result, the output attends to #a objects in total. We utilize the attentions in unique operations named BIND and UNBIND. Recall that the predicates in the effects can refer only to the specified action parameters. Therefore, we limit the dynamics to the objects attended by x (Fig. 3). We iteratively extracts the sub-axes of z/n attended by x using matrix operations (with an abuse of notation) z†/n = (x)nz/n ((B#a×O)nBO
n...O×P/n → B#a n...#a×P/n). We call it BIND(z/n,x), as it binds the parameters X to the values in objects O, similar to the function call in a typical programming language. It is also similar to applying numpy.take in all dimensions. Binding z/n for all arities results in BIND(z,x) = (BIND(z/1,x), · · · BIND(z/N,x)). We also define UNBIND(z†/n,x) = (x>)nz†/n ∈ BO
n...O×P/n which restores the original shape, but fills the cells with zeros for the objects not attended by x.
Example 1. For a moment let’s ignore that we use a boolean tensor and let z/2 = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] with O = {1, 2, 3} which represents a two-arg function f(2, 1) = z/22,1 = 4 etc. When we bind z/2 to (2, 1) by x = [[.01, .98, .01], [.98, .01, .01]], we obtain z†/2 = x(z/2)x> ≈ [[5., 4.], [2.1, 1.1]] ≈ [[5, 4], [2, 1]], approximating the subspace extraction. To unbind it, x>(z†/2)x ≈ [[1.1, 2.1, .03], [3.9, 4.9, .09], [.05, .07, .01]] ≈ [[1, 2, 0], [4, 5, 0], [0, 0, 0]].
After the subspace extraction with BIND, we use APPLY and ACTION to find the dynamics / action model inside the subspace. Notice that ACTION and APPLY no longer directly refers to each object oi by the index i because BIND already resolves the index reference. They take flattened binary vectors of size ∑ n(#a) nP/n as the input / output.
In order to reflect the changes applied to the bounded representation to the whole representation, we use UNBIND: z∼i,1 = zi,0−UNBIND(zi,0† )+UNBIND(z ∼i,1 † ). UNBIND is unique in that the unattended axes have near-0 values (0 at the limit of Gumbel-Softmax annealing in PARAMS). Therefore, the propositions not bound by x retain their values from zi,0.
Finally, we replace SimpleFOSAE’s MLP decoder with NLM layers. To match the output shape with the input o ∈ RO×F , the final layer has F features and we use only the unary part of the result tuple, NLMF,σ(·)/1 = COMPOSE(·, 1, F, σ). The total loss is the sum of `(oi,0,o∼i,0), `(oi,1,o∼i,1), `(oi,1,o∼
i,1 ), `(zi,1, z∼i,1), and `(zi,1† , z ∼i,1 † ), plus the variational losses for discrete VAEs. Hyperpa-
rameters, choice of loss functions and the tuning process are detailed in the appendix Sec. C.
The number of parameters in FOSAE++ is not affected by the number of objects, since essentially each NLM layer performs a 1D pointwise convolution over tuples of objects.

heading: 4 EXPERIMENTS
text: We prepared 3 artificial environments: Photo-realistic Blocksworld (Asai, 2018) renders an RGB Blocksworld image with Blender 3D engine and extracts the bounding boxes and the image patches of each object. MNIST 8-Puzzle (Asai & Fukunaga, 2018) is a 42x42 pixel, monochrome imagebased version of the 8-Puzzle. Tiles contain hand-written digits (0-9) from the MNIST database (LeCun et al., 1998), and valid moves swap the “0” tile with a neighboring tile, i.e., the “0” serves as the “blank” tile in the classic 8-Puzzle. Sokoban (Culberson, 1998) is a PSPACE-hard puzzle domain whose visualization was obtained from the PDDLGym library (Silver & Chitnis, 2020). In all datasets, segmentation is already provided by a domain-specific code, but in principle it can be replaced by the output of object-detectors such as YOLO (Redmon & Farhadi, 2018). The detail of the data preparation is available in the appendix Sec. D. Fig. 4 shows some comparisons between the input oi,0 and the reconstruction o∼i,0. In Sokoban, FOSAE++ generates up to 63 objects in a single scene.

heading: 4.1 INTERPOLATION/EXTRAPOLATION TO VARYING, UNSEEN NUMBER OF OBJECTS
text: The key characteristics of the FOSAE++ is that once trained, its network structurally generalizes to the different number of objects. We demonstrate this performance in interpolation/extrapolation tasks, where in the former we randomly drop a certain number of objects from the input, and in the latter, we use an environment with a different distribution or more objects. In both tasks, FOSAE++ shows an excellent reconstruction for a varying, untrained number of objects (Fig. 5).

heading: 4.2 PLANNING EXPERIMENTS
text: Finally, we ran Fast Downward Helmert (2006) planning system with LMcut Helmert & Domshlak (2009) heuristics and A∗ search configuration on the PDDL domains generated by FOSAE++. For the planning to succeed, FOSAE++ must be accurate not only about the direct reconstruction
o∼i,0,o∼i,1 but also about the dynamics that predicts the successor state z∼i,1 and its reconstruction o∼ i,1.
Due to the time constraint, we only performed the experiments for 8-Puzzle as of now. From the fixed goal state (the solved state of the puzzle), we performed a domain-specific Dijkstra search and sampled the initial states optimally L-steps away. Each initial/goal states are rendered, cropped into object-based input, then encoded by FOSAE++ to produce the symbolic initial/goal states. The planning results are visualized and manually inspected/validated by us. The symbolic classical planner correctly produced a solution which can be visualized into correct state transitions. See Sec. E.1 for the visualization of all 20 instances.

heading: 5 RELATED WORK
text: James et al. (2020a;b) build on existing work (Konidaris et al., 2014; 2015; 2018) to find an egocentric logical representation that is invariant to the state of the observer in a complex environment, or an object-centric, property-based logical representation. Both representations are special cases of First-Order Logic representation where all predicates are unary. These approaches also require a training input that already contains action symbols (e.g., toggle-door). Andersen & Konidaris (2017) obtains an MDP-based PPDDL (Probabilistic PDDL) model using Active Learning. Incorporating Active Learning in FOSAE++ for self data collection is future work.
Huang et al. (2019) reported that direct discretization approach performed worse than an approach that plans in the probabilistic representation obtained by neural networks. However, the question of continuous versus discrete is not conclusive. They used a naive rounding-based discretization which may cause a significant amount of information loss compared to the state-of-the-art discrete variational autoencoders. Furthermore, they rely on mappings from raw inputs to hand-coded symbolic representations that require supervised training.
Object-based input we used can be obtained from state-of-the-art object-recognition methods such as YOLO (Redmon & Farhadi, 2018). More recent systems (Greff et al., 2019; Engelcke et al., 2020) can handle shapes not limited to rectangles, and can be trained unsupervised. Connecting our network with these systems is an exciting avenue of future work.
Eslami et al. (2016) and Xu et al. (2019) proposed an attention-based and a Bayesian approach to the problem of variable-sized object reconstruction. Their work do not address the state dynamics and is orthogonal to our work. The primary difference from our approach is that they studied on sequentially storing and retrieving a variable amount of information into/from a fixed-sized array, while we store them in a latent representation of the corresponding size.

heading: 6 CONCLUSION
text: We proposed a first fully neural First-Order Logic Action Model Acquisition system that successfully produces a lifted PDDL planning model from noisy, segmented images alone. The system generates three types of symbols (action, predicate, and propositional) without manual labeling, and provides STRIPS-compatible explanations of actions. The learned results are generalized over the objects, thus can extrapolate to different and dynamically changing environments. We demonstrated that the state-of-the-art classical planner can produce correct solutions (as theoretically guaranteed) only with the interactions with the learned model. Note that the classical planner we used has no learning components. Our results partially support the Physical Symbol Systems Hypothesis by Newell & Simon (1976): In our word, a formal high-level representation learned from the raw inputs is sufficient for intelligent actions. Future work includes extensions to the formalisms with higher expressivity, including Probabilistic PDDL (Younes & Littman, 2004), Axioms (Thiébaux
et al., 2005), ADL (Pednault, 1987), numeric planning (Fox & Long, 2006), or Hierarchical Task Networks (Erol et al., 1994; Nau et al., 2003; Nejati et al., 2006).

heading: A EXTENDED BACKGROUNDS

heading: A.1 DISCRETE VARIATIONAL AUTOENCODERS
text: Variational AutoEncoder (VAE) is a framework for reconstructing an observation x from a compact latent representation z that follows a certain prior distribution. Training is performed by maximizing the sum of the reconstruction loss and the KL divergence between the latent random distribution q(z|x) and the target distribution p(z), which gives a lower bound for the likelihood p(x) (Kingma & Welling, 2013). While typically p(z) is a Normal distribution N (0, 1), Gumbel-Softmax (GS) VAE (Jang et al., 2017) and Binary-Concrete (BC) VAE (Maddison et al., 2017) use a discrete, uniform categorical/Bernoulli(0.5) distribution, and further approximate it with a continuous relaxation by introducing a temperature parameter τ that is annealed down to 0.
We denote corresponding activation function in the latent space as GS(l) and BC(l), where l and l each represents a logit vector and scalar. A latent vector z ∈ [0, 1]C of GS-VAE is computed from a logit vector l ∈ RC by z = GS(l) = SOFTMAX((l + GUMBEL(0, 1))/τ), where C is the number of categories, GUMBEL(0, 1) = − log(− logu) and u ∼ UNIFORM(0, 1) ∈ [0, 1]C . A latent scalar z of BC-VAE is computed from a logit scalar l ∈ R by z = BC(l) = SIGMOID((l + LOGISTIC(0, 1))/τ), where LOGISTIC(0, 1) = log u− log(1− u) and u ∼ UNIFORM(0, 1) ∈ R. Both functions converge to discrete functions at the limit τ → 0: GS(l) → arg max(l) (in one-hot representation) and BC(l) → STEP(l) = (l < 0) ? 0 : 1. Typically, GS-VAE and BC-VAE contains multiple latent vectors / latent scalars to model the complex input.
There are more complex variations such as VQVAE van den Oord et al. (2017), DVAE++Vahdat et al. (2018b), DVAE# Vahdat et al. (2018a). They may contribute to the stable performance, but we leave the task of faster / easier training to the future work.

heading: A.2 NEURAL LOGIC MACHINE
text: The NLM (Dong et al., 2019) is a neural Inductive Logic Programming (ILP) (Muggleton, 1991) system based on First-Order Logic and the Closed-World Assumption (Reiter, 1981). Given a set of base predicates grounded on a set of objects, NLMs sequentially apply horn rules to draw further conclusions such as a property of or a relationship between objects. For example, in Blocksworld, based on a premise such as (on a b) for blocks a, b, NLMs can infer (not (clear b)). NLM has two unique features: (1) The ability to combine the predicates of different arities, and (2) size invariance & permutation equivaliance on input objects, which is achieved by enumerating the permutations of input arguments.
NLM is designed to work on hand-crafted binary tensors representing propositional groundings of FOL statements. The format of these binary tensors are the multi-arity representation (z/1, . . . ,z/N) identical to the latent space of SimpleFOSAE.
NLM is designed for a subset of First Order Logic where every statement is a horn rule, contains no function terms (such as a function that returns an object), and all rules are applied between neighboring arities. With these assumptions, the statements fall in one of the three types below:
(expand) ∀x#p; p(X;x#p)← p(X), (reduce) p(X)← ∃x#p; p(X;x#p),
(compose) q(X)← F (⋃ π (P ∪ P ∪ P )/#q(π(X)) ) .
Here, p, p, p, q ∈ P, P , P ,Q (respectively) are predicates, X = (x1, . . .) is a sequence of parameters, F(T ) is a formula consisting of {∧,∨,¬,>,⊥} and T , (P ∪ P ∪ P )/#q is a set of predicates of the same arity as q, and π(X) is a permutation of X , which is used for composing the predicates with the different argument orders.
All three rules can be implemented as tensor operations (Fig. 6). Given a binary tensor z/n of shape O n. . .O × P/n, expand copies the n-th axis to n + 1-th axis resulting in a shape On+1. . . O × P/n, and reduce takes the max of n-th axis resulting in a shape On−1. . . O × P/n. While the original paper
proposed both min and max versions to represent ∀ and ∃, with enough number of layers only one of them is necessary because ∀x; p(·, x) = ¬∃x;¬p(·, x). Finally the COMPOSE combines the information between the neiboring tensors z/n, z/n−1, z/n+1. In order to use the information in the neighboring arities (P , P and P ), the input concatenates z/n with EXPAND(z/n−1) and REDUCE(z/n+1) (→ O n. . .O×C where C = P/n+ P/n−1 + P/n+1). Next, a PERMUTE function enumerates and concatenates the results of swapping the first n axes in the tensor (→ O n. . .O×(!n·C)). It then applies a n-D pointwise convolutional filter fn withQ output features (→ O n. . .O × Q). In the actual implementation, this n-D pointwise filter is merely a 1D convolution performed on a matrix reshaped intoOn×(!n·C). It is activated by any nonlinearity σ to obtain the final result, which we denote as COMPOSE(z, n,Q, σ). Formally, ∀j ∈ 1..n,∀oj ∈ 1..O,
COMPOSE(z, n,Q, σ)o1···on = σ(fn(Πo1···on)) ∈ RQ where Π = PERMUTE ( EXPAND(z/n−1); z/n; REDUCE(z/n+1) ) ∈ BO n...O×(!n·C)
An NLM contains N (the maximum arity) COMPOSE operation for the neighboring arities, with appropriately omitting both ends (0 and N + 1) from the concatenation. We denote the result as NLMQ,σ(z) = (COMPOSE(z, 1, Q, σ), · · · , COMPOSE(z, N,Q, σ)). This horizontal arity-wise compositions can be layered vertically, allowing the composition of predicates whose arities differ more than 1 (e.g., 2 layers of NLM can combine unary and quaternary predicates).
Two minor modification was made from the original paper. First, we use a slightly different tensor shapes: For the notational conveniency, we use hypercube-dimensional tensors of shape BO n...O×P/n, instead of the original formulation BO×O−1×...O−n−1×P/n which tries to reduce the size by disallowing the duplicates in the parameters. Our modification does not significantly affect the complexity of the representation because the original representation also has O(On) complexity.
Second, we do not use the nullary predicates z/0 in order to disallow VAEs from encoding environment-specific information in it.

heading: A.3 LEARNING DISCRETE LATENT DYNAMICS USING BACK-TO-LOGIT
text: APPLY(ai, zi,0) is an arbitrary MLP, i.e., a neural black-box function that does not directly translates to a STRIPS action model, preventing efficient search with state-of-the-art classical planner. CubeSpace AE (Asai & Muise, 2020) addresses this issue by Back-To-Logit technique which replaces the MLP. Back-to-Logit places a so-called cube-like graph prior on the binary latent space / transitions. To understand the prior, the background of cube-like graph is necessary.

heading: A.3.1 CUBE-LIKE GRAPHS AND ITS EQUIVALENCE TO STRIPS
text: Asai & Muise (2020) identified that state transition graphs of STRIPS planning problems form a graph class called directed cube-like graph (Payan, 1992). A cube-like graph is a simple2 undirected graph G(S,D) = (V,E) defined by sets S and D. Each node v ∈ V is a finite subset of S, i.e., v ⊆ S. The set D is a family of subsets of S, and for every edge e = (v, w) ∈ E, the symmetric difference d = v ⊕ w = (v \ w) ∪ (w \ v) must belong to D. For example, a unit cube is a cube-like graph because S = {x, y, z}, V = {∅, {x}, . . . {x, y, z}}, E = {(∅, {x}), . . . ({y, z}, {x, y, z})}, D = {{x}, {y}, {z}}. The set-based representation can be alternatively represented as a bit-vector, e.g., V ′ = {(0, 0, 0), (0, 0, 1), . . . (1, 1, 1)}. We denote a one-to-one |S|-bit vector assignment as f : V → V ′ = {0, 1}|S|. In STRIPS modeling, we use a directed version of this graph class. For every edge e = (v, w) ∈ E, there is a pair of sets d = (d+, d−) = (w \ v, v \ w) ∈ D which satisfies the asymmetric difference w = (v \ d−) ∪ d+. It is immediately obvious that this graph class corresponds to the relationship between binary states and action effects in STRIPS, s′ = (s \ DEL(()a)) ∪ ADD(()a). As a special case, in an undirected STRIPS planning problem where all actions are reversible (i.e., for any action a such that t = a(s), there is an action a−1 that is able to reverse the effect s = a−1(t)), the state space graph is equivalent to an undirected cube-like graph.

heading: A.3.2 EDGE CHROMATIC NUMBER OF CUBE-LIKE GRAPHS AND THE NUMBER OF ACTION
text: SCHEMA IN STRIPS
Edge coloring of cube-like graphs provides an intuitive understanding of the action schema expressible in STRIPS formalism. Consider coloring a graph which forms a unit cube (Fig. 8) and whose node embeddings correspond to the latent space of some images. A cube-like graph on the left can be efficiently colored (i.e., by fewer colors) by the difference between the neighboring embeddings. Edges can be categorized into 3 labels (0, 0,⊕1), (0,⊕1, 0) and (⊕1, 0, 0) (6 labels if directed), where each label is assigned to 4 edges which share the same node embedding differences, as depicted by the upward arrows with the common node difference (0, 0,⊕1) in the figure. The set of node embedding differences corresponds to the set D, and each element of D represents an action, e.g., when the node embeddings are decoded into images, moving toward positive direction of xaxis may result in the “0” tile in the image moving to the right, and moving toward positive direction of z-axis may result in the “2” tile in the image moving to the right — effectively making actions compositional. In contrast, the graph on the right has node embeddings that are randomly shuffled. Despite having the same topology and the same embedding size, this graph lacks the common patterns in the embedding differences like we saw on the left, thus cannot be efficiently colored by the node differences.
2No duplicated edges between the same pair of nodes
As such, cube-like graphs can be characterized by the edge chromatic number (minimum edge coloring) according to the node differences. We now discuss some theoretical properties of the graph coloring with and without the assumptions on the colors and the node differences. Theorem 1 (Edge chromatic number Vizing (1965)). Let the edge chromatic number c(G) of an undirected graph G be the number of colors in a minimum edge coloring. Then c(G) is either ∆ or ∆ + 1, where ∆ is the maximum degree of the nodes.
Theorem 2. Mapping E → D provides an edge coloring and thus c(G) ≤ minf |D|.
Proof. f is one-to-one: w 6= w′ ⇐⇒ f(w) 6= f(w′). For any set X , f(w) 6= f(w′) ⇐⇒ X ⊕ f(w) 6= X ⊕ f(w′). For any adjacent edges (v, w) and (v, w′), w 6= w′ because G is simple (at most one edge between any nodes), thus f(v) ⊕ f(w) 6= f(v) ⊕ f(w′). The remainder follows from the definition.
For Thm. 2, equality holds for hypercubes. For a certain embedding dimension |S|, there are graph instances where c(G) < minf |D| (Fig. 9, left). We “proved” this with an Answer Set Programming solver Potassco Gebser et al. (2011).
Based on these results, we next consider the minimum number of actions required to model an undirected cube-like graph G(S,D) as a planning model. We restrict ourselves to a preconditionfree planning domains in order to focus on the action effects. We first need a lemma in order to focus on undirected graphs. Lemma 1. Let P be a precondition-free grounded STRIPS planning problem which contains irreversible actions. There is a corresponding planning problem P ′ whose state transition graph is identical to that of P and whose precondition relaxation P ′′ is reversible.
Proof. For any irreversible action a in P , add a set of actions A−1(a) whose size is |A−1(a)| = 2|ADD(()a)∪DEL(()a)|. Each action a′ ∈ A−1(a) contains those effects which encode one of the possible non-deterministic outcomes of reversing a, and contains an unsatisfiable precondition (e.g., adding a new proposition whose value is constantly false). Then the state transition graphs of P and P ′ are identical and P ′′ is reversible.
Since we assume precondition-free domains, for any P we could instead consider P ′′ to discuss the effects of reversible planning domains. Let G be an undirected cube-like graph which is isomorphic to the state transition graph of a precondition-free reversible planning model P .
Theorem 3. Let Pc be another planning problem definition which models G. The action effects in Pc are allowed to use conditional effects. Then the minimum number of actions in Pc required to model G is c(G).
Proof. For each color c ∈ 1..c(G) and for each edge (v, w) colored as c, and for each propositional value f(w)i ∈ {0, 1}, we add a conditional effect to c-th action. If f(w)i = 0, add a delete-effect, and f(w)i = 1, add an add-effect. The effect is conditioned by the full conjunction of f(v) using negative preconditions. See Fig. 9 (right), where all effects are put in one conditional effect.
Theorem 4. The minimum number of actions required to model G without conditional effects (i.e., by STRIPS effects) is minf 2|D|.
Proof. Each d ∈ D needs 2 actions for forward/backward directions.
Thm. 3 indicates that conditional effects can compact as many edges as possible into just A = ∆ or ∆ + 1 actions regardless of the nature of the transitions, while Thm. 2 and Thm. 4 indicate that STRIPS effects require a larger number of actions. Therefore, merely assigning action symbols to state transitions (which is equivalent to edge coloring) does not result in a compact STRIPS model. Notice that the vanilla MLP AAE binds an unrestricted edge chromatic number c(G) = ∆ or ∆ + 1 by the maximum number of action labels A (a hyperparameter of the network), but does not bind 2|D|, the edge chromatic number in terms of neighboring node embedding differences. In order to find a compact STRIPS action model, we should instead bind 2|D| by A and restrict latent state transitions to follow a STRIPS transition rule.
In order to constrain the learned latent state space of the environment to a cube-like graph, we propose Cube-Space AutoEncoder. We first explain a vanilla Space AutoEncoder, an architecture that jointly learns the state and the action model by combining the SAE and the AAE into a single network. We then modify the APPLY progression to restrict state transitions. Due to the flexibility of neural networks, the loss enhanced by the restriction automatically propagates to the state representation, i.e., it modifies a state representation in order to reduce the loss produced by the restricted action model.
A.3.3 VANILLA SPACE AUTOENCODER
The vanilla Space AutoEncoder (Fig. 10, right) connects the SAE and AAE subnetworks. The necessary modification, apart from connecting them, is the change in loss function. The original AAE was trained to optimize the distance between the binary vectors using binary crossentropy (BCE), which is assymmetric in definition: BCE(x, y) = x log y + (1− x) log(1− y). While this was not problematic in the AAE which uses the fixed state representation x and the successor prediction y, it is more natural for Vanilla Space AE to use a symmetric loss that equally affects x and y.
In addition to the loss for the successor prediction in the latent space, we also ensure that the predicted successor state can be decoded back to the correct image oi,1. Thus, the total loss is a sum of the losses for: (1) the main reconstructions `(oi,0,o∼i,0) and `(oi,1,o∼i,1), (2) the successor latent state reconstruction `(zi,1, z∼i,1), (3) the image reconstruction from the predicted successor `(oi,1,o∼i,1), and (4) the KL regularization. We call the second term as direct loss.
We next formally analyze this training objective. Given an observed transition (o0,o1), we assume that o0 followsN (o∼0, σ0) and o1 followsN (o∼ 0 , σ1) (See Sec. 2.2 for explanation). The maximization objective is a log-likelihood of observing a pair of states o0,o1.
We iteratively derive the lower bounds (ELBO) by inserting several latent variables. We first introduce z0, z1:
log p(o0,o1) ≥−DKL(q(z0, z1 | o0,o1)||p(z0, z1)) + Eq(z0,z1|o0,o1)[log p(o0,o1 | z0, z1)]. (1)
The first term of Eq. 1 is the KL divergence for z0, z1. Since all latent variables are assumed to be independent (mean-field assumption), p(z0, z1) = p(z0)p(z1), where p(z0), p(z1) are respectively the prior distributions for the binary latent variable z0 and z1 that we discussed in Sec. ??. q(z0, z1 | o0,o1) can also be decomposed because, in the encoder modeled by q, z0 depends only on o0 and z1 depends only on o1. Therefore, the entire KL divergence is divided into individual KL divergence:
DKL(q(z 0, z1 | o0,o1)||p(z0, z1)) =DKL(q(z0 | o0,o1)||p(z0)) +DKL(q(z1 | o0,o1)||p(z1))
=DKL(q(z 0 | o0)||p(z0)) +DKL(q(z1 | o1)||p(z1)). (2)
We next decompose the second term in Eq. 1 as shown in Eq. 3. This derivation is possible because o∼0 and o∼1 are generated independently in the decoder (the network that produces o∼0,o∼1 from z0, z1) modeled by p. The first term in Eq. 3 corresponds to a reconstruction loss for o∼0 (MSE due to p(o0 | z0) = N (o∼0, σ0)).
log p(o0,o1 | z0, z1) = log p(o0 | z0, z1) + log p(o1 | z0, z1) = log p(o0 | z0) + log p(o1 | z0, z1). (3)
Next, we derive the lower bound of the second term in Eq. 3 by introducing a one-hot categorical latent variable a for action labels, and its prior distribution p(a | z0, z1) = Cat(1/A) (uniform categorical distribution of A categories).
log p(o1 | z0, z1) ≥−DKL(q(a | o1, z0, z1)||p(a | z0, z1)) + Eq(a|o1,z0,z1)[log p(o1 | a, z0, z1)]. (4)
Finally, to complete the Vanilla Space AE network, we should model the second term in Eq. 4 using another latent variable z∼1. We further derive the lower bound using the same reformulation:
log p(o1 | a, z0, z1) ≥−DKL(q(z∼1 | o1,a, z0, z1)||p(z∼1 | a, z0, z1)) + E
q(z∼1|o1,a,z0,z1)[log p(o 1 | z∼1,a, z0, z1)]. (5)
The second term of Eq. 5 is a reconstruction loss for o∼1 which can be computed by assuming p(o1 | z∼1,a, z0, z1) = N (o∼1, σ1). However, what should we use as a prior distribution p(z∼1 | a, z0, z1) in the first term (KL divergence)? We set it to be q(z1 | o1), because the choice of a prior is arbitrary and we want that the distribution of the predicted successor state z∼1 to be identical to the distribution of the successor state z1 directly encoded from the input. As a result, the total maximization objective is derived as follows:
log p(o0,o1) Interpretation
≥−DKL(q(z0 | o0)||p(z0)) KL divergence for z0 in Eq. 2 −DKL(q(z1 | o1)||p(z1)) KL divergence for z1 in Eq. 2 + log p(o0 | z0) Reconstruction loss `(o0,o∼0) in Eq. 3 −DKL(q(a | o1, z0, z1)||p(a | z0, z1)) KL divergence for a in Eq. 4 −DKL(q(z∼1 | o1,a, z0, z1)||q(z1 | o1)) KL divergence between z1 and z∼1 in Eq. 5
= direct loss `(z1, z∼1)
+ log p(o1 | z∼1,a, z0, z1). Reconstruction loss `(o1,o∼1) in Eq. 5 (6)
Direct loss `(z1, z∼1) can be computed by the same method introduced in (Sec. ??). We assume q(z∼1 | o1,a, z0, z1) = Bernoulli(q∼1) and q(z1 | o1) = Bernoulli(q1) where q∼1 = σ(l ∼1 ), q1 = σ(l1), and l ∼1
, l1 are the inputs to the corresponding Binary Concrete activation. Then the KL divergence is computed as
DKL(q(z ∼1 | o1,a, z0, z1)||q(z1 | o1)) = DKL(Bernoulli(q∼1)||Bernoulli(q1))
= q∼1 log q∼1 q1 + (1− q∼1) log 1− q ∼1 1− q1 . (7)
Coincidentally, this is DKL(q(z ∼1 | o1, z0,a)||Bernoulli(0.5)) + log 2 + BCE(q∼1, q1), whose last binary cross entropy term similar to the loss function of AAE.
Implementation note: If we instead optimize DKL(q(z∼1 | o1, z0,a)||Bernoulli(0.5)) + log 2 + BCE(z∼1, z1), the training becomes slower due to the unnecessary logistic noise inside BC of z∼1 = BC(l ∼1 ) and z1 = BC(l1).
Implementation note: It is important to bootstrap various terms in the optimization objective. In our experience, delaying the application of the direct loss until a certain epoch seems crucial, and we suspect the reason is due to the relatively short network distance between the two latent spaces zi,1/z∼i,1 compared to oi,1/o∼i,1. If we enable the direct loss from the beginning, the total loss does not converge because z∼i,1 prematurely converges to zi,1 causing a mode-collapse (e.g. all 0), before the image en/decoder learns a meaningful latent representation.

heading: A.3.4 CUBE-SPACE AE
text: Cube-Space AE modifies the APPLY network so that it directly predicts the effects without taking the current state as the input, and logically computes the successor state based on the predicted effect and the current state.
A naive implementation of such a network is shown in Fig. 11 (left). The EFFECT network predicts a binary tensor of F × 3 using F -way Gumbel-Softmax of 3 categories. Each Gumbel Softmax corresponds to one bit in the F -bit latent space and 3 classes correspond to the add effect, delete effect and NOP, only one of which is selected by the one-hot vector. The effects are applied to the
current state either by a max/min operation or its smooth variants (smooth min/max). Formally, the naive Cube-Space AE is formulated as follows:
BF 3 z∼i,1 = APPLY(zi,0,ai) = min(max(zi,0, ADD(()ai)), 1− DEL(()ai)) or (8) smin(smax(zi,0, ADD(()ai)), 1− DEL(()ai)), where (9)
EFFECT(()ai) = GS(MLP(ai)) ∈ BF×3, ADD(()ai) = EFFECT(()ai)0, DEL(()ai) = EFFECT(()ai)1, NOP(a i) = EFFECT(()ai)2,
smax(x, y) = log(ex + ey), smin(x, y) = − log(e−x + e−y). (10)
While being intuitive, we found these naive implementations extremely difficult to train. Our contribution to the architecture is Back-to-Logit (BtL, Fig. 11, right), a generic approach that computes a logical operation in the continuous logit space. We re-encode a logical, binary vector back to a continuous, logit representation by an monotonic function m. This monotonicity preserves the order between true (1) and false (0) even after transformed into real numbers. We then apply effects by adding the continuous state vector with a continuous effect vector. The effect vector is produced by applying an MLP named EFFECT to the action vector ai. After adding the continuous vectors, we re-activate the result with a discrete activation (Binary Concrete). Formally,
z∼i,1 = APPLY(zi,0,ai) = BC(m(zi,0) + EFFECT(()ai)). (11)
We found that an easy and successful way to implement m is Batch Normalization Ioffe & Szegedy (2015), a method that was originally developed for addressing the covariate shift in the deep neural networks.
Additional background: For simplicity, we consider a scalar operation, which can be applied to vectors element-wise. During the batch training of the neural network, Batch Normalization layer BN(x) takes a minibatch input B = { x1 . . . x|B| } , computes the mean µ and variance σ2, then shift
and scale each xi so that the result has a mean of 0 and a variance of 1. It then shifts and scales the results by two trainable coefficients γ and β. Formally,
∀xi ∈ B; BN(xi) = x i − µ σ2 γ + β. (12)
While rescaling the normalized result by γ and β seem to negate the original purpose of normalization, the presense of normalization to mean 0 / variance 1 is crucial. Notice that the first scaling depends on other training examples in the same minibatch, while γ and β is not dynamically adjusted for the minibatch. For example, imagine two minibatches B1 and B2, where B1 accidentally tends to contain larger values than B2 but the variance within B1 and B2 are the same. The bias is canceled by the normalization, and thus the outputs rescaled by γ and β are computed based on a relative scale inside the corresponding minibatch.
Implementation note: Since ai is a probability vector overA action ids and ai eventually converges to a one-hot vector due to Gumbel-Softmax annealing, the additional MLP can be merely a linear
embedding, i.e., EFFECT(()ai) = Eai, where E ∈ RF×A and z ∈ BF . It also helps the training if we apply BatchNorm on the effect vector. Therefore, a recommended implementation is
APPLY(a, zi,0) = BC(BN(zi,0) + BN(Eai)) (13)
where EFFECT(()ai) = BN(Eai).

heading: A.3.5 BACK-TO-LOGIT AND ITS EQUIVALENCE TO STRIPS
text: States learned by BTL has the following property:
Theorem 5. (Asai & Muise, 2020) Under the same action a, state transitions are bitwise monotonic, deterministic, and restricted to three mutually exclusive modes, i.e., for each bit j:
(add:) ∀i; (zi,0j , z i,1 j ) ∈ {(0, 1), (1, 1)} i.e. z i,0 j ≤ z i,1 j (14)
(del:) ∀i; (zi,0j , z i,1 j ) ∈ {(1, 0), (0, 0)} i.e. z i,0 j ≥ z i,1 j (15)
(nop:) ∀i; (zi,0j , z i,1 j ) ∈ {(0, 0), (1, 1)} i.e. z i,0 j = z i,1 j (16)
This theorem guarantees that each action deterministically sets a certain bit on and off in the binary latent space. Therefore, the actions and the transitions satisfy the STRIPS state transition rule s′ = (s\DEL(()a))∪ADD(()a), thus enabling a direct translation from neural network weights to PDDL modeling language.
The proof is straightforward from the monotonicity of the BatchNorm and Binary Concrete. Note that we assume BatchNorm’s additional scale parameter γ is kept positive or disabled.
Proof. For readability, we omit j and assumes a 1-dimensional case. Let e = EFFECT(()a) ∈ R, which is a constant for the fixed input a. At the limit of annealing, Binary Concrete BC becomes a STEP function, which is also monotonic. BN is monotonic because we assumed the scale parameter γ of BN is positive, and the main feature of BN also only scales the variance of the batch, which is always positive. Then we have
zi,1 = STEP(BN(zi,0) + e). (17)
The possible values a pair (zi,0, zi,1) can have is (0, 0), (0, 1), (1, 0), (1, 1). Since both STEP and BN are deterministic at the testing time (See Ioffe & Szegedy (2015)), we consider the deterministic mapping from zi,0 to zi,1. There are only 4 deterministic mappings from {0, 1} to {0, 1}: {(0, 1), (1, 1)}, {(1, 0), (0, 0)}, {(0, 0), (1, 1)}, {(0, 1), (1, 0)}. Thus our goal is to show that the last mapping {(0, 1), (1, 0)} is impossible in the latent space { . . . (zi,0, zi,1) . . . } produced by BTL.
To prove this, first, assume (zi,0, zi,1) = (0, 1) for some index i. Then
1 = STEP(BN(0) + e)⇒ BN(0) + e > 0⇒ BN(1) + e > 0⇒ ∀i; BN(zi,0) + e > 0. (18)
The second step is due to the monotonicity BN(0) < BN(1). This shows zi,1 is constantly 1 regardless of zi,0, therefore it proves that (zi,0, zi,1) = (1, 0) cannot happen in any i.
Likewise, if (zi,0, zi,1) = (1, 0) for some index i,
0 = STEP(BN(1) + e)⇒ BN(1) + e < 0⇒ BN(0) + e < 0⇒ ∀i; BN(zi,0) + e < 0. (19)
Therefore, zi,1 = 0 regardless of zi,0, and thus (zi,0, zi,1) = (0, 1) cannot happen in any i.
Finally, if the data points do not contain (0, 1) or (1, 0), then by assumption they do not coexist. Therefore, the embedding learned by BTL cannot contain (0, 1) and (1, 0) at the same time.

heading: A.4 EFFECT RULE EXTRACTION
text: To extract the effects of an action a from Cube-Space AE, we compute ADD(a) = APPLY(a,0) and DEL(a) = 1 − APPLY(a,1) for each action a, where 0,1 are vectors filled by zeros/ones and has the same size as the binary embedding. Since APPLY deterministically sets values to 0 or 1, feeding
these vectors is sufficient to see which bit it turns on and off. For each j-th bit that is 1 in each result, a corresponding proposition is added to the add/delete-effect, respectively.
In FOSAE++, we extracts the effects from parameter-bounded subspace zi,0† , z i,1 † . The representation is a tuple z† = (z†/1 · · · z†/N), where z†/n ∈ B#a× n...×#a×P/n. BTL then operates on flattened
and concatenated binary vectors of size ∑ n #a
nP/n: The input, the output, and the effect share this shape. We extract the effects from this BTL vector in the same manner as noted above. After the extraction, however, each bit is converted to a lifted predicate according to the position. For example, when a bit that corresponds to z†/21,2,5 has turned from 0 to 1, then the add-effect contains p5(?arg1, ?arg2), where ?arg1 is a parameter used in the lifted PDDL encoding.
We show an example of such a learned PDDL model below, obtained from 8-Puzzle with P/1 = P/2 = 333 (reformatted for readability). Note that we disabled the nullary predicates P/0 and z/0, which consumes the first 333 dimensions in the flattened vector. Another note is that we also count the number of appearances of each action in the training dataset. If an action label is never used in the dataset, it is not exported in the resulting PDDL output. Thus, the index for the action starts from 7 in the example.
(define (domain latent) (:requirements :strips :negative-preconditions) (:predicates (p333 ?x0) ... (p665 ?x0) (p666 ?x0 ?x1) ... (p998 ?x0 ?x1))
(:action a7 :parameters (?x0) :precondition (and (p339 ?x0) (p388 ?x0) (p391 ?x0) (p398 ?x0) (p402 ?x0)
(p420 ?x0) (p421 ?x0) (p446 ?x0) (p447 ?x0) (p473 ?x0) (p475 ?x0) (p489 ?x0) (p491 ?x0) (p502 ?x0) (p516 ?x0) (p559 ?x0) (p588 ?x0) (p615 ?x0) (p641 ?x0) (p648 ?x0) (p831 ?x0 ?x0) (p950 ?x0 ?x0) (not (p333 ?x0)) (not (p371 ?x0)) (not (p375 ?x0)) (not (p388 ?x0)) (not (p402 ?x0)) (not (p406 ?x0)) (not (p421 ?x0)) (not (p447 ?x0)) (not (p454 ?x0)) (not (p504 ?x0)) (not (p508 ?x0)) (not (p519 ?x0)) (not (p524 ?x0)) (not (p526 ?x0)) (not (p562 ?x0)) (not (p584 ?x0)) (not (p593 ?x0)) (not (p617 ?x0)) (not (p640 ?x0)) (not (p652 ?x0)) (not (p824 ?x0 ?x0)) (not (p892 ?x0 ?x0)) (not (p926 ?x0 ?x0)) (not (p975 ?x0 ?x0)) (not (p994 ?x0 ?x0)))
:effect (and (p349 ?x0) (p361 ?x0) (p366 ?x0) (p370 ?x0) (p371 ?x0)
(p378 ?x0) (p381 ?x0) (p385 ?x0) (p388 ?x0) (p401 ?x0) (p408 ?x0) (p421 ?x0) (p432 ?x0) (p445 ?x0) (p454 ?x0) (p475 ?x0) (p491 ?x0) (p496 ?x0) (p502 ?x0) (p503 ?x0) (p504 ?x0) (p507 ?x0) (p517 ?x0) (p526 ?x0) (p550 ?x0) (p562 ?x0) (p563 ?x0) (p575 ?x0) (p584 ?x0) (p588 ?x0) (p599 ?x0) (p601 ?x0) (p607 ?x0) (p612 ?x0) (p617 ?x0) (p631 ?x0) (p640 ?x0) (p641 ?x0) (p647 ?x0) (p656 ?x0) (p663 ?x0) (p724 ?x0 ?x0) (p768 ?x0 ?x0) (p831 ?x0 ?x0) (p902 ?x0 ?x0) (p911 ?x0 ?x0) (p993 ?x0 ?x0) (not (p339 ?x0)) (not (p355 ?x0)) (not (p365 ?x0)) (not (p391 ?x0)) (not (p397 ?x0)) (not (p398 ?x0)) (not (p402 ?x0)) (not (p406 ?x0)) (not (p422 ?x0)) (not (p446 ?x0)) (not (p447 ?x0)) (not (p448 ?x0)) (not (p451 ?x0)) (not (p456 ?x0)) (not (p472 ?x0)) (not (p473 ?x0)) (not (p478 ?x0)) (not (p489 ?x0)) (not (p490 ?x0)) (not (p495 ?x0)) (not (p516 ?x0)) (not (p518 ?x0)) (not (p524 ?x0)) (not (p525 ?x0)) (not (p527 ?x0)) (not (p534 ?x0)) (not (p544 ?x0)) (not (p559 ?x0))
(not (p561 ?x0)) (not (p615 ?x0)) (not (p624 ?x0)) (not (p629 ?x0)) (not (p642 ?x0)) (not (p646 ?x0)) (not (p651 ?x0)) (not (p653 ?x0)) (not (p720 ?x0 ?x0)) (not (p813 ?x0 ?x0)) (not (p824 ?x0 ?x0)) (not (p892 ?x0 ?x0)) (not (p894 ?x0 ?x0)) (not (p926 ?x0 ?x0)) (not (p931 ?x0 ?x0)) (not (p975 ?x0 ?x0)) (not (p994 ?x0 ?x0))))
(:action a22 :parameters (?x0) :precondition ...

heading: A.5 PRECONDITION LEARNING WITH DYNAMICS REVERSED IN TIME
text: In the main text, we simplified the model by showing only the forward dynamics, i.e., the dynamics in the same direction as the time. This forward dynamics can model the effects (add/delete) of the actions. However, the forward dynamics is insufficient for learning the preconditions of the actions. The original CSAE paper (Asai & Muise, 2020) used an add-hoc method that extracts common bits of the current states.
In contrast, we added a network that uses the same BTL mechanism that is applied backward in time, i.e., predict the current state zi,0 from a successor state zi,1 and a one-hot action vector ai. We named the network REGRESS(zi,1,ai), alluding to the regression planning (Alcázar et al., 2013) literature.
In REGRESS, add-effects and delete-effects now correspond to positive preconditions and negative preconditions. A positive precondition (normal precondition) requires that a proposition is > prior to using an action. In contrast, a negative precondition requires that a proposition is ⊥ prior to using an action. While negative preconditions are (strictly speaking) out of STRIPS formalism, it is commonly supported by the modern classical planners that participated in the recent competitions. To extract the preconditions from the network, we can use the same method used for extracting the effects from the progressive/forward dynamics.
The entire model thus looks as follows.
(encoder) zi,0, zi,1 = ENCODE(oi,0), ENCODE(oi,1)
(action parameters, uses NLMs) xi = PARAMS(zi,0, zi,1)
(parameter-bound subspace extraction) zi,0† , z i,1 † = BIND(z i,0,xi), BIND(zi,1,xi)
(action assignment) ai = ACTION(zi,0† , z i,1 † )
(bounded forward dynamics) z∼i,1† = APPLY(z i,0 † ,a i)
(bounded backward dynamics) z∼i,0† = REGRESS(z i,1 † ,a i)
(reflection to global forward dynamics) z∼i,1 = zi,0 − UNBIND(zi,0† ,x i) + UNBIND(z∼i,1† ,x i)
(reflection to global backward dynamics) z∼i,0 = zi,1 − UNBIND(zi,1† ,x i) + UNBIND(z∼i,0† ,x i)
(reconstructions) o∼i,0,o∼i,1 = DECODE(zi,0), DECODE(zi,1)
(reconstruction based on forward dynamics) o∼i,1 = DECODE(z∼i,1)
(reconstruction based on backward dynamics) o∼i,0 = DECODE(z∼i,0)
The total loss is `(oi,0,o∼i,0)+`(oi,1,o∼i,1)+`(oi,1,o∼i,1)+`(oi,0,o∼i,0)+`(zi,1, z∼i,1)+`(zi,1† , z ∼i,1 † )+ `(zi,0, z∼i,0) + `(zi,0† , z ∼i,0 † ) + Reg.
B IMPLEMENTATION DETAIL
We based our code on a publicly-available Latplan source code repository (https://github. com/guicho271828/latplan/), which is based on Keras Deep Learning library. The repository hosts its own Genetic Algorithm based hyperparameter tuner which we mention several times later.
B.1 INPUT DATA FORMAT AND LOSS FUNCTIONS
As we discussed in the earlier appendix section, our final loss function consists of 9 terms: `(oi,0,o∼i,0)+`(oi,1,o∼i,1)+`(oi,1,o∼ i,1 )+`(oi,0,o∼
i,0 )+`(zi,1, z∼i,1)+`(zi,1† , z ∼i,1 † )+`(z i,0, z∼i,0)+
`(zi,0† , z ∼i,0 † ) + Reg. We first describe the input data format that is shared among different domains, then the loss functions defined on it.
Our input/output format o ∈ RO×F consists of O objects (environment-dependent) each having F features. F features consists of image-based features and the coordinate / dimension-based features (Fig. 12). All image patches extracted from the observations are resized into a fixed height H , width W and color channel C = 3. Each flattened object vector has the size F = H ×W × C + 4. The last 4 dimensions contain the center coordinate and the actual height/width before the resizing. Out of the 9 terms in the total loss, the terms that apply to the object vectors of this form are `(oi,0,o∼i,0), `(oi,1,o∼i,1), `(oi,1,o∼i,1), `(oi,0,o∼i,0).
For this data format, the loss function consists of the mean square value of the image part, and the square sum of the coordinate / dimension parts. We do not average the losses for the coordinates/dimensions to avoid making the gradient minuscule. To further enhance this direction, we additionally have the coordinate loss amplifier λ tuned by GA, as it is often the case that the object location has more visual impact on the reconstruction. Note that, for the tuning with the validation set and the evaluation with the test set, we set λ = 1 in order to obtain the consistent, comparable measuring. λ is altered only during the training. Formally, for the i-th objects in the input o and the reconstruction o∼, we define the loss as follows. These losses are averaged over the objects and the batch dimensions.
`(oi,o ∼ i) =
1
HWC ||oi,1..HWC − o ∼ i,1..HWC ||22 + λ||oi,HWC..F − o ∼ i,HWC..F ||22
We call the remaining losses except the regularization terms as “latent dynamics loss”. They operate on the binary latent data activated by BinaryConcrete: `(zi,1, z∼i,1), `(zi,1† , z ∼i,1 † ), `(z
i,0, z∼i,0), `(zi,0† , z ∼i,0 † ). However, note that during the training, all values used for the loss calculation are still continuous due to BinaryConcrete’s annealing. This means we can’t use Binary Cross Entropy BCE, the standard loss function for binary classifications, because the “training data” is also a noisy probability value. It is also in fact symmetric — as we discussed in the previous appendix sections, the role of these losses is not only just to obtain the accurate dynamics, but also to shape the state representation toward a cube-like graph. While there are several candidate loss functions that can be considered, we adapted Symmetric Cross Entropy (Wang et al., 2019) designed for noisy labels, which simply applies BCE in both ways. Formally, given BCE(z , z∼) = − ∑ i (zi log z ∼ i + (1− zi) log(1− z ∼ i)),
`(z , z∼) = BCE(z , z∼) + BCE(z∼, z).
The remaining regularization losses include KL divergence for Discrete VAEs, as well as the L1 regularization for the latent vectors zi,0, zi,1 which were proven useful in Asai & Kajino (2019).
Finally, we define the magnitude and the warmup of each loss. The magnitude multiplies each loss and is tuned by the GA tuner. Similar to λ, the values are set to 1 during the evaluation. We have α for the L1 regularization, β for KL divergence, and γ for the latent dynamics loss.
The warmup mechanism works by setting these values to 0 until the training reaches a certain epoch defined by the ratio r relative to the total number of epochs. We used the warmup rα, rγ for α, γ, as well as rrec and the rdyn for the main reconstruction loss `(oi,0,o
∼i,0) + `(oi,1,o∼i,1) and the dynamics-based reconstruction loss `(oi,1,o∼i,1) + `(oi,0,o∼i,0).
The motivation behind these magnitudes and warmups is to balance the speed of convergence of the various parts of the networks. Depending on the hyperparameters (depth, width of the layers), occasionally the network completely ignores the dynamics, falling into something similar to (but the mechanism will be very different from) a mode collapse where the effect caused by the dynamics is empty, e.g., the forward dynamics produces the same state as the current state.
Another failure mode of the network is that the dynamics loss is too strong, due to its BCE which could become too large compared to the reconstruction loss. As a result, the network learns a perfect but meaningless latent space dynamics which does not produce correct reconstructions. Tuning the warmup and balancing the losses addressed these issues.

heading: B.2 NETWORK DETAIL
text: FOSAE++ consists of 6 networks: ENCODE, PARAMS, ACTION, APPLY, REGRESS, DECODE. Note that BIND and UNBIND are weight-less operations.

heading: B.2.1 ENCODER
text: ENCODE can be divided into trivial continuous feature extraction phase (pre-encoder) and the actual FOSAE encoder. The feature extractor is a simple 1D pointwise convolution over the objects, i.e., same as applying the same Dense/Fully-Connected(FC) layer on each individual object. Its depth and the hidden layer width is tuned by the GA tuner. All activations except the last Binary Concrete are Rectified Linear Unit (Fukushima, 1980; Nair & Hinton, 2010). The architecture is illustrated in Fig. 12.
We should note that we assign the same number of predicates to different arities: P/1 = P/2 . . . = P/N . In this Network Detail section, we denote P = P/1 . . . = P/N as a hyperparameter that specifies the number, although P was already used in the main text as the total number of predicates (i.e., ∑ n P/n).

heading: B.2.2 PARAMS
text: PARAMS consists of multiple NLM layers, as depicted in Fig. 14. All hidden activations are ReLU, and the width Q of PARAMS is tuned by the GA tuner.
z/1 z/2
multi-arity predicate representation
z/3
multi-arity continuous
representation
/1 /2 /3
/1 /2 /3 NLM-relubn-dropout(0.1) NLM
OxP OxOxP OxOxOxP shape: OxQ OxOxQ OxOxOxQ
Ox#a OxOx#a
OxOxOx#a transpose
O
#a-way GumbelSoftmax
#a arg1 arg2 arg3
Continuous Logit
arg1 arg2 arg3
objects being attended#axO
Figure 14: PARAMS network using NLMs.

heading: B.2.3 ACTION, APPLY / REGRESS
text: Since ACTION takes parameter-bounded representations (zi,0† , z i,1 † ) whose objects are already selected and appropriately reordered by BIND, we simply apply an MLP whose last layer has an output size A, and is activated by Gumbel-Softmax. This results in selecting one action, represented by a one-hot vector. Before applying the MLP, we should flatten and concatenate the tuple of vectors, each of size #a × n. . . × #a × P . The depth and the hidden layer width of the MLP is automatically tuned.
For APPLY and REGRESS, we similarly flatten the input and directly apply the BTL structure described in the earlier sections. The EFFECT network in z∼i,1 = BC(BN(zi,0) + EFFECT(ai)) is a single linear layer without bias combined with a batch normalization. In other words, with the weight matrix W ,
z∼i,1 = BC(BN(zi,0) + BN(Wai)).

heading: B.2.4 DECODER
text: The decoder consists of NLMs followed by a post-decoder that shares the same width and depth as the pre-encoder. In Blocksworld, we additionally made the decoder a Bayesian Neural Network to absorb the uncertainty in the reconstruction. Details are available in (Sec. D.2).

heading: B.3 HYPERPARAMETER TUNER
text: The tuning system assumes that the hyperparameter configuration is a vector/dict of categorical/finite candidates. The tuner is a textbook integer GA with uniform crossover and point mutation. Assume that a hyperparameter configuration is represented by H values. A new configuration p = {p1, · · · , pH} is created from two parents q = {q1, · · · , qH} and r = {r1, · · · , rH} by ∀i; pi = RandomChoice(qi, ri), and then a single value is randomly mutated, i.e., for m = RandomChoice(1..H), pm ← RandomChoice(ValidValuesOf(pm) \ {pm}). It stores all evaluated configurations and never re-evaluates the same configuration.
In the beginning of the tuning process, it bootstraps the initial population with a certain number of configurations. New configurations are evaluated by the validation loss, then pushed in a sorted list. A certain number of best-performing configurations in the list are considered as a “live” population. In each iteration, it selects two parents from the live population by inverse weighted random sampling of the score, preferring the smaller validation losses but also occasionally selecting a second-tier parent. Non-performing configurations will “die” by being pushed away from the top as the algorithm finds better configurations. The evaluation and insertion to the queue is asynchronous and all processes can run in parallel.

heading: C TRAINING DETAILS AND HYPERPARAMETERS
text: We trained multiple network configurations on a distributed compute cluster equipped with Tesla K80 and Xeon E5-2600 v4, which is a rather old hardware. The list of hyperparameters are shown in Table 3.
In all experiments, we used the total of 5000 state transitions (10000 states) from the training environments. The details of data collection for each domain is available in the later sections. This dataset is divided into training/validation/test sets (90%:5%:5%). The Genetic Algorithm hyperparameter tuner uses the validation loss as the evaluation metric.
We set a limit of 1500 total runs for each environment, with 100 initial population, and ran maximum 100 processes in parallel for each environment. As an additional trick, to avoid testing unpromising candidates (e.g., those with diverging loss), the epoch parameter is forced to be 50 in the first 100 configurations and the runs finish quickly. The rest of the runs use these initial populations as the parents, but replaces the epoch with an appropriate value selected from the candidates.

heading: C.1 REPRODUCIBILITY, TRAINING CURVES ON 3 RUNS
text: In Fig. 15, we show 3 training runs on each environment with the same hyperparameter configuration found by our tuner. It reproduces an almost identical, stable curves across 3 runs.

heading: D DOMAIN-WISE DETAILS

heading: D.1 SOKOBAN
text: We generated 10000 transitions of each training problem using Dijkstra search from the initial state. We shuffled the 50000 transitions, subsampled 5000 transitions out of 50000, then stored them in a single archive. The rendering and other data are obtained from PDDLGym library Silver & Chitnis (2020). Each tile is resized into 16x16 pixels, and the tile ordering is also shuffled.
In order to make the training data dimension consistent and make it convenient for GPU-based training, we performed a so-called Random Object Masking which removes a certain number of objects from each state via random selection. The idea is similar to masking-based image augmentation commonly used in the computer vision literature, but the purpose is different from those: Ours has more emphasis on having the consistent number of objects in each state. For example, Sokoban training problem 0 (leftmost in Fig. 16) has 49 tiles, while the problem 1 (the second picture) has 72 tiles. In the combined archive, the number of objects is set to the smallest among the problems.
We also removed certain tiles that cannot be reached by the player. For example, in problem 0, the three floors on the top left corner cannot be reached by the player. Similarly in problem 1, the bottom right corner is not reachable by the player. We performed a simple custom reachability analysis using the meta-information obtained from PDDLGym library. This helped reducing the dataset size and thus the training.
Finally, during the dataset merging, we accounted for the potential location bias caused by the map size difference. For example, if we preserved the original x, y locations, the tiles tend to be biased around 0, 0 and the location around (x, y) = (12, 12) (by tiles) is never occupied by any tile. To address the issue, for each state pair, we relocated the entire environment by a value selected uniformly randomly from a certain width. The width is decided from the maximum dimension of all training problems, i.e., 12x12. For example, a state pair in problem 4 (which has a 10x9 map dimension) will be shifted by a random value between 0..(12− 10) in x-axis, and 0..(12− 9) in y-axis.

heading: D.2 BLOCKSWORLD
text: We generated 5000 random state transitions using Photorealistic-Blocksworld dataset Asai (2018), which in turn is based on CLEVR Johnson et al. (2017) dataset generator. It uses Blender 3D rendering engine to produce realistic effects such as metallic reflection, surface materials and shadows. The objects are extracted from the information available from the generator metadata. We cropped the image region suggested by the metadata, resized them into 32x32x3 image patches and stored them in an archive. The size of the objects reported by the metadata may vary and is noisy due to the camera jitter, object location, and the heuristics used for extracting the objects. This is a case even for the objects that are not moved by the random actions.
Each transition is generated by sampling the current state and then randomly moving a block. To sample a state, we first generate a set of block configurations (color, material, shape, size), then placing them randomly on a straight line in the 3D environment without collisions. When we move a block, we select a set of blocks of which nothing else is on top, choose one randomly, pick a new horizontal location and place it at the lowest non-colliding height. We ensure that the block is always moved, i.e., not stacked on top of the same block, and not on the floor if it is already on the floor.
In Blocksworld, we noticed that a same conceptual symbolic action (such as move) may have a nondeterministic outcome in the image space while each individual concept is discrete and deterministic.
For example, move may relocate a block onto a floor, but the resulting position is chosen randomly during the dataset generation, i.e., it can be placed anywhere on the floor.
To model this uncertainty in our framework, we used a Bayesian Neural Network layers in the decoder for Blocksworld domain. The final output o ∈ RO×F is produced by two NLMs of output feature size F , each producing the mean and the variance vectors µ, ∈ RO×F , and the output reconstruction is generated by the random sampling: o = µ+σ · , where is a random noise vector following normal distributionN (0, 1). During the testing (e.g., visualization), the random sampling is disabled and we use the value µ for rendering.
D.3 8-PUZZLE
The 8-puzzle domain generator is directly included in the Latplan code base. It uses MNIST dataset for its tiles and the tile size is set to 16. Similarly, we generated 5000 random state transitions using the generator.

heading: E PDDL GENERATION AND PLANNING EXPERIMENTS
text: E.1 EXAMPLE OUTPUT: 8-PUZZLE

Review summaries:
This work presents FOSAE++, an end-to-end system capable of producing "lifted" action models provided only bounding box annotations of image pairs before and after an unknown action is executed. Building on recent work in the space, the primary contribution of this work is to generate PDDL action rules. To accomplish this, the authors introduce novel 'params' function that use the Gumbel-Softmax function to implement a differentiable mechanism for selecting which entities are relevant to the current action and feeds them into the new 'bind' and 'unbind' functions that select those elements in the tensor predicting their relevance. Overall, this work is a meaningful contribution in the direction of generated lifted action models without direct labeled data.

This paper addresses the problem of learning dynamics model directly from raw sensory inputs. The authors propose an unsupervised end-to-end model that can perform high-level tasks planning on raw observations. This work extends Asai et al. 2020, 2019 etc, and with improved symbol generation and lifted PDDL. The authors follow the experimental setup as seen in prior work, where three artificial environments (blocksworld, MNIST 8-puzzle, and sokoban) are used for planning.