Learning Neuro-Symbolic Skills for Bilevel Planning
Authors: Tom Silver
Ashay Athalye
Joshua B. Tenenbaum
Tomás Lozano-Pérez
Leslie Pack Kaelbling
Venue: CoRL 2022 Poster (2022)
Keywords: Skill Learning
Neuro-Symbolic
Task and Motion Planning

Abstract: Decision-making is challenging in robotics environments with continuous object-centric states, continuous actions, long horizons, and sparse feedback. Hierarchical approaches, such as task and motion planning (TAMP), address these challenges by decomposing decision-making into two or more levels of abstraction. In a setting where demonstrations and symbolic predicates are given, prior work has shown how to learn symbolic operators and neural samplers for TAMP with manually designed parameterized policies. Our main contribution is a method for learning parameterized polices in combination with operators and samplers. These components are packaged into modular neuro-symbolic skills and sequenced together with search-then-sample TAMP to solve new tasks. In experiments in four robotics domains, we show that our approach --- bilevel planning with neuro-symbolic skills --- can solve a wide range of tasks with varying initial states, goals, and objects, outperforming six baselines and ablations.

Learning Neuro-Symbolic Skills for Bilevel Planning
Tom Silver, Ashay Athalye
Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling MIT Computer Science and Artificial Intelligence Laboratory ftslvr, ashay, jbt, tlp, 1pk Onit eemit.edu
Abstract: Decision-making is challenging in robotics environments with con- tinuous object-centric states, continuous actions, long horizons, and sparse feed- back. Hierarchical approaches, such task and motion planning (TAMP), ad- dress these challenges by decomposing decision-making into two or more lev- els of abstraction. are given, prior work has shown how to learn symbolic operators and neural samplers for TAMP with manually designed parameterized policies. Our main contribution is method for learning parameterized polices in combination with operators and samplers. These components are packaged into modular neuro- symbolic skills and sequenced together with new tasks. experiments in four roboties domains, we show that our ap- proach - bilevel planning with neuro-symbolic skills can solve a wide range of tasks with varying initial states, goals, and objects, outperforming six base- lines and ablations. Video: https://youtu. wwt.ttt...bpo/PPPPGG Code: https: //tinyurl. cwm/sooolll/ceeeccc @kill-learning
a setting where demonstrations and symbolic predicates
TAMP to solve
Keywords: Skill Learning, Neuro-Symbolic, Task and Motion Planning
Introduction
Decision- Decision-making in robotics environments with continuous state and action spaces is especially challenging when tasks have long horizons and goal-based objectives. A common strategy is to
decompose decision-making into a high level ("what to do"") and low level ("how to do it")) [1, 3]. Seminal work by Konidaris et al. [4] considers such hierarchy where symbolic Al planning [5] is used to sequence together cemporally-extendae method for learning symbols from known skills. In this work, we study the inverse: learning skills from known symbols [7, 8, 9, 10, 11]. We are motivated by cases where it is easier to design [12,
continuous skills [6]. Their main contribution is a
13, 14, 15, 16] or learn [17, 18, 19, 20, 21) symbols than it is to design skills. In the long term, envision a continually learning robot that uses symbols to learn skills and vice versa.
discrete relations between objects. A of pred- icates induces state abstraction [3, 22] of a continuous object-centric state space. For example, consider the Stick Button environment (Figure 1, third column), where robot must press buttons either with its gripper, or by using a stick as tool. Predicates include Grasped and Pressed, in- ducing an abstract state such as (Grasped (robot, stick), Pressed (button1),.... }. We lever- age symbolic predicates to learn skills that generalize substantially from limited demonstration data. Concretely, we learn skills that achieve certain symbolic effects (c.g.. Grasped (robot, stick)) from states where certain symbolic preconditions hold (e.g., HandEmpty (robot)).
We consider symbols in the form of predicates
Symbolic scaffolding for skill learning has two major benefits. First, by drawing on Al planning techniques, we can efficiently chain together long sequences of learned skills [4, 5, 10]. This ben- efit is key in our setting where planning the low-level transition space is practically infeasible, even though a black-box transition function is deterministic and known - actions are continuous, solutions can exceed 100 actions, and there is no extrinsic feedback available before a task goal reached. Second, because each skill acts in a confined region of the state space, learning is easier than it would be for monolithic goal-conditioned policy (e.g., with behavioral cloning [23]).
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Cover
Doors
Stick Button
Coffee
Train Taoles
Evaluation Tacles
Figure 1: Environments. Top row: train task examples. Bottom row: evaluation task examples. See ($6) for descriptions of each environment.
While symbols offer useful structure for skill learning and planning, they also present challenges. In particular, symbolic state abstractions often lossy [12, 13, 24]. in that they discard information that may be important for decision-making like the reachability of buttons or the relative stick grasp in Stick Button. This lossiness be mitigated some degree by inventing new predicates [4, 21J, but in robotics environments, there are often kinematic and geometric constraints that are hard to perfectly abstract away [1, 25, 26]. These difficulties lead us the following key desiderata.
Key Desideratum (KD1), Because the abstraction is lossy, some environment states that corre- spond the same symbolic state may be better than others terms of completing task. Thus, skill should be able to reach many different environment states ("subgoals") ") that correspond to the same abstract state. For example, a skill that achieves Grasped (robot, stick) should not always lead to the same relative grasp: the grasp should be lower when buttons are farther away, and higher collisions with the gray stick holder would prevent a bottom grasp. This desideratum implies that lookahead may be necessary, where the choice of skill early in the plan should be influenced by soft or hard constraints later in the plan. also implies that we should not rely on the (abstract) subgoal condition in the framework of Konidaris et al. [4].
Key Desideratum (KD2). When the state abstraction is lossy, it is not always possible to take actions in the environment that correspond to particular abstract state sequence. Thus, an agent should be able to consider multiple skill sequences that reach the same goal from the same initial abstract state. For example, without a predicate that reflects button reachability, the agent may first plan to press all buttons without the stick, but some buttons are unreachable, will need the stick.
To address these two key desiderata, we propose an approach for learning and planning with newro- symbolic skills. A neuro-symbolic skill consists of a symbolic operator [27], a neural subgoal- conditioned policy, and neural subgoal sampler [28, 29]. The subgoals produced by the sampler and consumed by the policy correspond to different environment states that are consistent with the abstract effects of the operator, addressing KDI. The symbolic operators enable a bilevel planning algorithm [30, 31, 32], with Al planning in an outer loop and sampling in an inner loop. If sampling fails for operator sequence, the outer loop continues to another sequence, addressing KD2. Skills are learned from demonstrations without any prior knowledge about the number of skills.
Main Contributions. Prior works by Silver et al. [12] and Chitnis et al. [13] propose pipeline for learning operators and samplers for bilevel planning when given a set of parameterized policies'. While some policies may be human-designed and generally reusable, e.g., navigation policies that use motion planners, others are more comain-sppecfic. such as pouring a pot of coffee or opening a door (Figure 1). Our main contribution is method for learning these policies to complete the neuro- symbolic skills. We conduct experiments four robotics environments (Figure 1) that require a wide range of behaviors, such as pouring, opening, pressing, picking, and placing. We find that skills learned from 100-250 demonstrations generalize to novel states, goals, and objects. Furthermore, our approach bilevel planning with neuro-symbolic skills (BPNS) outperforms six baselines and ablations, including learning graph neural network-basc metacontrollers [15, 33].
'See (SC.3) for elaborated discussion the relationship between this work and [12, 13].

---

2
Problem Setting
We consider the problem of learning from demonstrations deterministic, fully-observed envi- ronments with object-centric states, continuous actions, and known transition function. An envi- roument is characterized by a tuple (,, f. V An object type À E A has a name (e.g.. button, robot) and tuple of real-valued features (e.g., (x, y, 2, radius, color, ...)) of dimension dim(A). An object o E O has a name (e.g., button1) and a type, denoted type(o) E A. A state I E X' is an assignment of objects to feature vectors, that is, x=o) E (edim((ype(o)) for o E O. The action space is denoted C R The transition function is known, deterministic mapping from state and action to a next state, denoted f : X' x U + X. A predicate 1 E Vp has a name (e.g., Touching) and a tuple of types (e.g., (stick, button)). A ground atom a predicate and a mapping from its type tuple to objects (e.g.. Touching(sti ck, button1)). A lifted atom instead has a mapping to typed variables, which are placeholders for objects (e.g.. Touching (?s, ?b)). Predicates induce state abstraction: abstract(z) denotes the set of ground atoms that hold true in x, with all others assumed false. We use E S to denote an abstract state, i.c., abstract X S.
An environment is associated with a task distribution T, where each I' E T is characterized by tuple (O, zo. 9, H.). The set of objects the task denoted O Importantly, objects vary between tasks. The initial state of the task is denoted 2g E X. The goal, a set of ground atoms with predicates in V and objects O, is denoted g. goal represents a set of abstract states: for example, there are many possible abstract states where for how many actions can be taken in a task, denoted H Z². A solution to a task a sequence of actions that achieves the goal and does not exceed the task horizon, i.e. that A S H, 9 C. abstract(ra), and x; f(x--1, Wz) for sisk k.
{Pressed (button2)) } holds. The task horizon, the limit
(wi,..., ug) such
We consider standard learning setting where train tasks drawn from from T are available at training time, and held-out evaluation tasks drawn from T are used for evaluation. Our objective to maximize the number of evaluation tasks solved within a wall-clock timeout. Each train task T is associated with one demonstration (T, प. Z), where
a solution for T and J = (zo, Fx3) is the sequence of states visited. In experiments, tasks are randomly sampled, and demonstrations are generated with handcrafted environment-specffe policies (but see (SC.4)).
3 Neuro-Symbolic Skills
How can we leverage the available demonstrations, predicates, and known transition model to max- imize the number of evaluation tasks solved hefore timeout? We propose learn and plan with neuro-symbolic skills. In this section, we define these skills formally; in ($4), we discuss planning; and in ($5), we address learning. See Figure 2 for a summary of the architecture.
A skill is tuple o = (V, w, n. σ) where is a tuple of arguments; w is an operator; т is a subgoal- conditioned policy; and a is subgoal sampler. A set of skills is denoted $. Arguments are variables with types in A and are shared by w. т, and a. An operator is tuple w = (u. E,E E-) where P, E+ E- are preconditions, add effects, and delete effects respectively, each set of lifted atoms over the predicates V and arguments A subgoal-conditionel policy is mapping π Ol% X x X + 2 that takes as input a tuple of objects for the arguments U. the current state x. and a subgoal state x', and maps it to action A subgoal sampler is generator of elements from a distribution
of subgoal states conditioned on object arguments and current state, that is, Ol7 x X + A(X) A ground skill is a skill with objects that match its arguments, denoted by a tuple ø = (ó. ō) where ō is tuple of objects in O with types matching U. (We use underlines to denote grounded entities.)
The corresponding ground operator, ground policy, and ground sampler are defined analogously,
Goal &
Al planner
Skills
Abstract state Ag
Operatorg
Absizuet state
Predicates
abstract
Sampler
Subgoal
abstract
Tasemedd
Initial state To
Poscy
Action o,
State 3,
Policy
Subgoo s'
Ground Skill
Figure 2: Architecture. Given learned symbolic operators, an AI planner is used to generate a candidate sequence of abstract states and skills. For each skill in the sequence, the learned sampler proposes specific subgoal state in the next abstract state for the learned policy to pursue.
3

---

tnst peesaetiectla
Dutton unresehable
sacclepity 1ea01 Fosaaed Jeaily
axponsaed
Collision with bolder
622 mmb 100
Pick
Safloely aic, kui
Rroceed (oucc)
Press button
with
Figure 3: Planning example. First, an abstract plan that directly presses both buttons fails. Later, an abstract plan that uses the stick initially fails due to collisions, but succeeds on the next sample.
each associated with objects that match their typed variables, denoted w (P, Et. E n. and respectively. A set ground skills t induces an abstract transition function F S x $ - S over the abstract state space through the add and delete effects of operators, given by F(s, Ø) (s) (T) E* defined only when P S (where is set subtraction).
The contract between skill's operator, policy, and sampler is the following: in states where the preconditions of ground skill hold, i... P abstract(z). the sampler should generate subgoal x' ~ æ(x) that satisfies the skill's effects, i.e. F(abstract(z)) Φ) = abstract(x'), and the actions proposed by the policy 2") should lead to x' from x. In practice, we initiate a ground skill only when the ground operator preconditions hold, and terminate at corresponding abstract successor state (cf. options [6]). Next, we show how to leverage this contract for efficient planning.
Bilevel Planning with Neuro-Symbolic Skills
Given skills and new task, we search for solution with bilevel planning. See Figure 3, Algo- rithm and [13] for extended description. Bilevel planning begins with outer loop (Line 2), where skill operators are used to generate up to Natstrcct abstract plans. An abstract plan for task = (O, 2o.9. H) is a sequence of ground skills that achieves the goal, i.e. Φ = (á,..... 2.) such that = abstract(zo), S st, e H + 1, and = F(s for S iss
Much work in the AI planning literature is de- voted efficiently generating these abstract plans, with most state-of-the-art techniques building on heuristic search. In experiments, we use A* search with the LMCut heuris- tic [34]. The Al planner details are not impor- tant for this work; we have also experimented with other planners and heuristics including GBFS, K [35]. FF (36), and various configu- rations of the Fast Downward planner [5]. Un- like typical Al planning, we continue the search after the first plan found, generating poten- tially infinite sequence of candidate plans (cf. top-k planning [35, 37, 38]).
(Tan(T, f)
// Params: Natetreet: H Nemples
8g f abstract(za)
// Uses operators
for @ in top-k(so. 9. Φ Naturact) do
7/ Uses samplers and policies
(efiieee T, $, Hain,, Namples) if not null then
// Planning succeeded return
// Planning failed
6
return null
Algorithm 1: Bilevel planning.
Given candidate abstract plan, we use the samplers and policies to perform a backtrack- the
ing search over action sequences (Line 3). For each ground skill in the abstract plan, ground sampler to produce a subgoal, and simulate the ground policy until either (I) the abstract transition is completed; or (2) maximum skill horizon Ham is reached. In case (1), continue to the next ground skill, or terminate if the goal is reached (Line 5), while in case (2) we backtrack up to N Namples times per step. If backtracking fails, continue the next abstract plan
This planning strategy is not probabilistically complete, but see [32, 39] for complete variations.

---

5 Learning Neuro-Symbolic Skills from Demonstrations
We now address the problem of learning skills & from a dataset of demonstrations D ( u. T. Recall that the data comprise raw sequences of states and actions for entire tasks. We preprocess the data in three steps: segmenting trajectories temporally, partitioning the segments into skill datasets, and lifting the objects within each skill dataset to variables. Then, for each skill dataset, we learn an operator, a subgoal-conditions policy, and subgoal sampler. We detail these steps below. This pipeline extends that of Chitnis et al. [13), who assume parameterized policies are given.
5.1 Data Preprocessing
Segmentation. For each demonstration T,, E D, we identify set of switch points 1 < iq < < ig < [.] Each pair of consecutive switch points (j, k) induces segment
- where yy = (u), uj+t,...., 24-1) 1) is a subsequence of actions from 7 and - 1::k is defined analo- gously. Temporal segmentation is a much studied problem in the literature [40, 41, 42]. In this work, inspired by the notion of contact-basee
modes (c.g.. [43]), we create a new switch point whenever there is change in the truth-value of any contact-related ground atom (SA). Intuitively, using more (fewer) predicates here instcad would lead to finer (coarser) temporal abstractions.
Partitioning. Next, we partition the set of all segments into a collection of skill datasets. We group segments into the same skill dataset if their abstract effects are equivalent up to object substitution. Formally, given segment y (ū.sk; Fy 1;k), let = abstract(z and s' = abstract(zs-11- (s' s, s \ s'). The affected object ser O comprises
The effects of the segment are (et ee all objects that appear in either et or e Two segments 71, 72 with effects (e)"eT)), (ej,e, and affected objects Of, Og are equivalent (71 72) if there exists injective mapping ò Of + Og such that d(e;) 63 and ((cT) = 62 and where each object in Of shares the type of the object it is mapped to in Og. The relation induces the partition: each partition contains equivalent segments.
Lifting. The segments in each skill dataset involve different objects. When we perform lifting, we replace the objects in a segment with variables that have the objects' types. For example, in Stick Button, the robot may use the stick to press buttoni in some segment, and the robot may use the stick to press button2 in another segment. After lifting, these objects would be replaced with ?robot, ?stich ck, and ?button. Formally, for each skill dataset D, for an arbitrary segment To E 2 with affected object set Og. we create one placeholder variable e for each object in Og. We order these variables arbitrarily into tuple U. Using the injective mappings d described above, it is then possible to map the affected object set for any segment in & to the variables J. In addition to aligning segments with different objects, variables define the skill scope: when computing abstract transitions, subgoals, and actions, the skills will reference only the object states for those variables. The final output of preprocessing is set of lifted skill datasets, i.e., skill datasets di, variables v, and mappings from segment objects to variables that we will use in skill learning, described next.
5.2 Skill Learning
Following previous work [13, 18, 21, 44], we use simple linear-time ap- proach to learn symbolic operators. We induce one operator from each lifted skill dataset o The variables e constructed via lifting comprise the operator arguments. The operator add and delete effects follow immediately from lifting a segment's effects (et,e") we replace all objects with the corresponding variables in G. to arrive lifted atom sets (E',, 1=+, To compute preconditions, we use all common lifted atoms in the initial segment states. For a segment 7 E & with states - let 8yy abstract(z;-1). and let S. denote the corresponding lifted atom set, with all affected ob- jects sy replaced with variables in v, and with any ground atoms in 8., involving objects not in T discarded. The preconditions P are then the intersection of lifted atom sets for each segment, denoted P = Ahee. s7. completing the operator w PT,,.E
Operator Learning.
Policy Learning. Recall the responsibility of a ground policy <(x, x') is to output an action u that will lead the agent closer to the subgoal z' from the state x. At this point in the pipeline, for each skill, we have available a dataset of subgoals being achieved: the final state of each segment in the skill dataset is a subgoal, and the preceding states and actions demonstrate behavior towards that subgoal. We can therefore use these data for supervised learning of a subgoal-conditioned policy. First, we use the skill scope to convert the segment states into -iedddiiensiionl vectors by concate-
5

---

nating the features of the objects in the scope together with the subgoal. Formally, for each state in each segment y E P we construct vector x" x(v1) c .... o x(,,), where (Di...., Un), a(1;)) = x((2) for the corresponding o; under lifting, and denotes vector concatenation. We then create a dataset of input-output pairs (x._1 o 1 u4) for j S < A in each segment with states Tz-1:k, where _x is the segment subgoal. We are now left with the problem of learning a policy from pector-basede supervised data. In this work, we use behavioral cloning with fully-connected neural networks (see ($B) for details), but many other choices are possible". At evaluation time, if the policy is grounded with objects = ( on), and queried in state x; then s<(1) o. ......r o.r(o,) is the state input to the neural network, while the subgoal input is generated by the ground sampler.
Sampler Learning. The role of the subgoal sampler is to propose different specific states, among the infinitely many consistent with the effects of the operator, for the policy to pursue. This flexibility important for KD1 ($1). To learn samplers, we can largely reuse the datasets from policy learning: for each (zoy, u) in the policy learning dataset, with y denoting the subgoal, we create a (z, y) input- output pair for sampler learning. Following [13], we learn two neural networks for each sampler. The first network outputs the mean and diagonal covariance of a Gaussian distribution. The second network is a binary classifier that takes a candidate (x. y) sampled from the Gaussian as input and accepts or rejects. These two networks are capable of modeling a richer class of distributions than the Gaussian alone, e.g., multimodal distributions. See ($B) for architecture and training details*.
We make two modifications to the approach described above to promote generalization. First, we use the relative subgoal 2 x.. for each step in the segment, rather than the absolute subgoal =x_1 (ef. goal relabelling [29, 46)). At evaluation time, we derive the absolute subgoal from the relative subgoal when the policy is first initialized, and update the relative subgoal after each transition. Second, we detect and remove subgoal dimensions that are static across all data (e.g., object mass),
Limitations. Our approach assumes that the given predicates comprise a useful state abstraction of the underlying state space with respect to the task distribution. With random or meaningless predi- cates, we would not expect to outperform non-symbolic baselines. However, previous work [12, 21] suggests that operator and sampler learning are robust to limited degree to predicate ablation or addition; we expect the same for policy learning'. We assume that effective behavior can be deter- mined from the skill scope alone and do not learn skills that handle variable number of objects. Our method would create a separate skill for each unique skill dataset arising from affected object sets of different sizes, which might be preferred in some scenarios (picking up one, vs. ten, vs. two-hundred coins off the ground) but not others (tying multiple sticks together with a piece of string). Relational neural networks (c.g.. GNNs) offer one possible path to address these two limitations. Through the lens of parameterized policy learning [47, 48, 49, 50], subgoals are just one form of parameteriza- tion among many possible alternatives. We also aspire to learn from human demonstrations on real robots; data efficiency results in the next section suggest that this is feasible.
6 Experimental Results
Our experiments address five questions about bilevel planning with neuro-symbolic skills (BPNS):
Q1. How many train tasks are required by BPNS effectively solve held-out evaluation tasks?
Q2. Does BPNS generalize to unseen numbers of objects?
Q3. Can BPNS learn skills to complement existing
purpose skills?
Q4. How important is the ability to sample multiple subgoals per abstract plan step (KDI)?
Q5. How important the ability to generate multiple candidate abstract plans (KD2)?
Environments. We describe environments here at high level, with details in (SA). The Cover environment was introduced by [12, 13, 21]. A robot must pick and place blocks cover target regions on 1D line. The robot can only initiate and release grasps in certain allowed regions. For example, if the robot grasps a block on the left side, it may not be able to place to completely cover a target, depending on the allowed regions. Tasks have two blocks and two targets. There is random variation in the initial poses of the blocks, targets, allowed regions, and robot gripper.
2 We also experimented briefly with implicit behavioral cloning [45] but did not notice any improvements. 'The policy and sampler for skill are trained separately. In preliminary experiments, we observed that training them jointly sometimes performs hetter training is stopped early, hut is otherwise similar. Seee ($C.5) and (SC.6) for results with superfluous predicates and objects.

---

Cover
Doors
Stick Button
Coffee
36 Evall Tacke Solved
100
90
8
60
1030
200
450
600
1800
200
400
# Demonstrations
*
Demonst rations
#
Demonstrations
Demonstrations
BPNS (Ours BPNS No Subgoal
GNN Meta
GNN Meta No Subgoal
Abstrmet Plans-1
Samples-
Figure 4: Main results. Evaluation task success rates versus number demonstrations. Lines are means and error bars are standard deviation over 10 random seeds. GNN BC is excluded because its performance consistently very poor, and its training is the most expensive; see the table below.
Doors is loosely inspired by the classic Four Rooms [6]. A robot must navigate target room while avoiding obstacles and opening doors. Doors have handles that require different rotations to open. In this environment, to address (Q3), skill that moves the robot between configurations using a motion planner (BiRRT) is provided; door opening must be learned. Tasks have 4-25 rooms with random connectivity. The goals, obstacles, initial robot pose, and door parameters also vary.
The Stick Button environment is described in ($1). 1). Train tasks have 1-2 buttons and evaluation tasks have 3-4 buttons. Tasks vary in the initial poses of the stick, holder, robot, and buttons.
In Coffee, robot must move a pot of coffee to a hot plate, turn on the hot plate by pressing a button, and then move pour the coffee into cups. The cups have different sizes and require different amounts of coffee. Depending on its orientation, the pot may need to be rotated before the handle can be grasped; the orientation is reflected in the predicates. Overfilling cup or pouring from too far results in a spill (failure). Train tasks have 1-2 cups and evaluation tasks have 2-3 cups. Tasks vary in the poses of the robot, pot, and cups, and in cup sizes. The goal is to fill all cups.
Approaches. We briefly describe the approaches we evaluate, with details in ($B).
BPNS: Our main approach, bilevel planning with learned neuro-symbolic skills.
BPNS No Subgoal: Our main approach, but with no samplers, and with subgoal-conditioned
policies replaced with regular policies z X - U. This approach is inspired by [7, 9, 91].
GNN Meta: Instead of Al planning, this approach learns graph neural network (GNN) meta- controller [9, 15, 33] that outputs next skill based on the current state, abstract state, and goal. GNN Meta No Subgoal: Same as GNN Meta, but without samplers, and with regular policies. This baseline is the closest existing methods the Al planning literature [9, 10, 51, 52].
GNN BC: Learns a monolithic coal-conditioned GNN policy by behavioral cloning (BC). Samples=1: An ablation of BPNS where N Nsamples during planning.
Abstract Plans=l: An ablation of BPNS where Nabstratt = 1 during planning.
All approaches are trained with the same demonstration data and use the same predicates. All approaches also use the transition function f. except for GNN BC, which is model-free.
Experimental Setup. For all environments and approaches, we use 50 evaluation tasks and 10 random seeds. The timeout for each evaluation task is 300 seconds and the horizon H 1000. Experiments were run on Ubuntu 18.04 using CPU cores of an Intel Xeon Platinum 8260 processor.
Results and Analysis. Figure 4 shows performance as function of demonstration count. BPNS performs well in all environments after 100-250 demonstrations (Q1). Stick Button is the most challenging because many considered abstract plans are infcasible, which can lead to timeouts. The strong performance in Stick Button and Coffee, which have more objects than seen during training, also allows us to affirmatively resolve (Q2). Strong performance in Doors, where motion planners are provided, confirms that BPNS is capable of incorporating ceneral-purpose skills (Q3).
The importance of generating multiple samples per abstract plan step (Q4) is reflected in the No Subgoal baselines and the Samples=1 ablation. The ablation performs worse than BPNS all environments, while the No Subgoal baselines perform worse in all except Doors with 1000 demon- strations. We found evidence for two explanations in our analysis. First, skill policies may fail to terminate with certain parameters, but succeed with others. Second, even if a policy terminates, the
7

---

agent may or may not be able to finish the rest of the task from that subgoal. These results confirm that KD1 ($1) is important for the strong performance of BPNS.
To analyze the importance of generating multiple candidate abstract plans (Q5), we can examine the GNN Meta baselines and the Abstract Plans=1 ablation. These approaches perform well in
Cover and Doors, where the first abstract plan is usually refinable into solution. In Coffee, the first
abstract plan does not rotate the pot, and therefore fails when the handle out of reach. See Figure
for example of failure in Stick Button. These results confirm the importance of KD2 ($1).
We also compare BPNS with GNN BC, a purely model- free approach. The table on the right reports the mean (standard deviation) percent of evaluation tasks solved per environment after training with 1000 demonstrations. We
Environment
BPNS (Ours)
GNN BC
Cover
(0.64)
4.60 (1.62)
Doces
98.80 (0.80)
9.20 (4.05)
Stick Button
(1.78)
0.20 (0.30)
Coffee
98.00
23.80 (4.76)
also experimented with fewer demonstrations for GNN BC and saw consistently poor performance. The failure of GNN BC is unsurprising given the limited data and the high degree of task variabilityy.
In the appendix, we present additional results that investigate the planning time of BPNS ($C.1), the poor performance of GNN Meta (&C.2), comparison to [12. 13] ($C.3), learning from human demonstrations (SC.4), the impact of irrelevant predicates ($C.5) or objects (SC.6), the filtering of low-data skills (SC.7), comparison to oracle skills ($C.8), the structure of the learned operators ($C.9), and skill learning with invented predicates (SC.10).
7 Related Work
Our work contributes to a long line of research in skill learning for robotics, with prior works con- sidering both reinforcement learning [53, 54, 55, 56, 57, 58, 59, 60, 61, 62] and learning from demonstrations [8, 40, 63, 64, 65, 66, 67, 68, 33J. In the context of hierarchical RL [28, 29, 69], it is possible to interpret our skill samplers as managers and policies as workers. Our work is also related to parameterized skill learning [47, 70, 48 71, 49, 50] if we view subgoals rameters. In the Al planning literature, prior works use a set of given planning operators to learn skills [7. 9, 10, 72, 73, 51, 52, 74]. Achterhold et al. [75] pursue similar approach in robotics setting. Recent work by Guan et al. [15] is closest to our contribution in that they use given operators to learn skills with latent parameters that can be used to reach diverse set of terminal states [61], and then learn a metacontoller over the skills. The "GNN Meta" haseline is inspired by this work. Instead of operators, recent works have also used pretrained language models to generate abstract "instructions" and then learn language-conditioned skills [68, 76, 77, 78].
pa-
Our work can also be viewed as learning components of a task and motion planning (TAMP) system. In this line, other work has concentrated on learning operators [79, 80, 81, 82, 44, 12, 83], learning samplers [39, 84, 85, 86, 13, 87], or learning predicates [17, 88, 4, 89, 90, 91, 18, 19, 20, 92, 21J. Our operator and sampler learning algorithms are most directly related to that of Chitnis et al. [13], who assume given parameterized policies. Driess et al. [26] use given operators to learn energy-based controllers from images in the context of constraint- based TAMP [43]. Wang et al. [86] learn to sample continuously parameterized skills to accomplish symbolic operator effects, such as pouring at certain angle to fill cup. To the best of our knowledge, our work is the first to learn operators, samplers, and policies for search-then-san -hen-sample TAMP [30, 31, 32].
8 Conclusion
In this work, we proposed bilevel planning with learned neuro-symbolic skills as a framework for recision-making in object-centric robotics environments. We confirmed experimentally that this ap- proach leads to strong generalization and data efficiency. There are many interesting open questions for future work. (1) Can neuro-symbollc skills be learned from invented predicates? In (SC. 10), we offer preliminary evidence suggesting the viability of this direction. (2) Can neuro-symbolic skills be learned with reinforcement learning? (3) Can latent object feature learning [16] be used to drive skill learning? Answering these questions will help us leverage hierarchical planning at scale.
8

---

Acknowledgements
We gratefully acknowledge support from NSF grant 1723381; from AFOSR grant FA9550- 1--550-17-1- 0165; from ONR grant -00014-18-1-2847; and from MIT-IBM Watson Lab. Tom is supported by an NSF Graduate Research Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors. We thank Rohan Chitnis, Nishanth Kumar, Willie B. McClinton, Aidan Curtis, Will Shen, and Anish Athalye for helpful comments earlier draft.
References
[1] R. Garrett, R. Chitmis, Holladay, B. Kim, T. Silver, L. Kaelbling, and Lozano-Peree Perez. Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4, May 2021.
[2] Bercher, R. Alford, and D. Höller. A survey on hierarchical planning-one abstract idea, many concrete realizations. In IJCAI, pages 6267-6275, 2019.
[3] L. Li, J. Walsh, and M. Littman. Towards unified theory of state abstraction for MDPs. International Symposium on Artificial Intelligence and Mathematics (ISAIM), 4(5):9, 2006.
[4] G. Konidaris, L. P. Kaelbling, and T. Jozano-Perez From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61: 215-289, 2018.
[5] M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191-246, 2006.
[6] R. S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for tem- poral abstraction in reinforcement learning. Artificial intelligence, 11 1999.
[7] M. R. Ryan. Using abstract models of behaviours to automatically generate reinforcement learning hierarchies. International Conference on Machine Learning (ICML), volume 2, pages 522-529. Citeseer, 2002.
[8] C. Paxton, F. Jonathan, M. Kobilarov, and G. D. Hager. Do what want, not what I did: Imi- tation of skills by planning sequences of actions. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3778-3785. IEEE, 2016.
[9] D. Lyu, F. Yang, B. Liu, and S. Gustafson. SDRL: interpretable and data-efficient deep rein- forcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 2970-2977, 2019.
[10] H. Kokel, A. Manobaran, S. Natarajan, B. Ravindran, and P. Tadepalli. RePReL: Integrating relational planning and reinforcement learning for effective abstraction. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 31, pages 533-541, 2021.
[11] Y. Wang. N. Figueroa, S. Li, A. Shah, and J. Shah. Temporal logic imitation: Learning plan- satisficing motion policies from demonstrations, 2022.
[12] T. Silver, R. Chitnis, J. Tenenbaum, L. Kaelbling, and T. Lozano-Pérez. Learning sym- bolic operators for task and motion planning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3182-3189 IEEE, 2021.
[13] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and I.. P. Kaelbling. Learning neuro- symbolic relational transition models for bilevel planning. 2022 IEFF/RSI International Con- ference on Intelligent Robots and Systems (IROS), 2022.
[14] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. Roboties: Science and Systems 2020 (RSS 2020). RSS Foundation, 2020.

---

[15] L. Guan, S. Sreedharan, and S. Kambhampati. Leveraging approximate symbolic models for reinforcement learning via skill diversity. arXiv preprint arXiv:2202.02886, 2022.
[16] C. Wang, D. Xu, and L. Fei-Fei. Generalizable task planning through representation pretrain- ing, arXiv preprint arXiv:2205.07990 2022.
[17] N. Jetchev, T. Lang, and M. Toussaint. Learning grounded relational symbols from continu- ous data for abstract reasoning. In Proceedings of the 2013 ICRA Workshop on Autonomous Learning, 2013.
[18] M. Asai and C. Muise. Learning neural-symbolic descriptive planning models via cube-space priors: the voyage home (to strips). In Proceedings of the Twenty-Ninth International Confer- ence on International Joint Conferences on Artificial Intelligence, pages 2676- 2682, 2021.
[19] A. Ahmetoglu, M. Y. Seker, Piater, E. Oztop, and E. Ugur. Deepsym: Deep symbol gen- eration and rule learning from unsupervised continuous robot interaction for planning. arXiv preprint arXiv:2012.02532, 2020.
[20] E. Umili, E. Antonioni, F. Riccio, R. Capobianco, D. Nardi, and G. De Giacomo. Learning a symbolic planning domain through the interaction with continuous environments. ICAPS PRI. Workshop, 2021.
[21] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, L. P. Kaclbling, and J. Tenen- baum. Inventing relational state and action abstractions for effective and efficient bilevel plan- ning. arXiv preprint arXiv:2203.096. 09634,, 2022.
[22] D. Abel, D. Hershkowitz, and M. Littman. Near optimal behavior via approximate state ab- straction. In International Conference on Machine Learning, pages 2915-2923, PMLR, 2016.
[23] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3(4):362-369) 2019.
[24] B. Marthi, J. Russell, and J. A. Wolfe. Angelic semantics for high-level actions. In Proceed- ings of the International Conference on Automated Planning and Scheduling (ICAPS), pages 232-239, 2007.
[25] M. Toussaint. Logic-geometric programming: An aptimization-based approach to combined task and motion planning. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
[26] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for long-horizon tasks from visual input. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14298 -1305,, 2021.
[27] R. E. Fikes and N. J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial intelligence, (3-4))189-2208 1971.
[28] P. Dayan and G. E. Hinton. Feudal reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 5, 1992.
[29] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[30] F. Gravot, S. Cambon, and R. Alami. aSyMov: planner that deals with intricate symbolic and geometric problems. In Robotics Research. The Eleventh International Symposium. Springer, 2005.
[31] S. Srivastava, E. Fang. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In International Conference on Robotics and Automation (ICRA), pages 636. IEEE, 2014.
[32] N. Dantam, 7. K. Kingston, S. Chaudhuri, and I.. F. Kavraki. Incremental task and motion planning: A constraint- bassee approach. In Robotics: Science and Systems, pages 1-6, 2016.
10

---

[33] Y. Zhu, P. Stone, and Y. Zhu. upttom-up skill discovery from unsegmented demonstrations for long- long-horizon robot manipulation. IEEE Robotics and Automation Letters, (12):4126-4133, 2022.
[34] M. Helmert and C. Domshlak. Landmarks, critical paths and abstractions: what's the difference anyway? In Nineteenth International Conference on Automated Planning and Scheduling (ICAPS), 2009.
[35] M. Katz, S. Sohrabi, O. Udrea, and D. Winterer. A novel iterative approach top-k planning. In Proceedings of the Twehty-Eighth International Conference on Automated Planning and Scheduling (ICAPS), 2018.
[36] Hoffmann. FF: The fast-forward planning system. magazine, 22(3):57-57, 2001.
[37] A. Riabov, S. Sohrabi, and O. Udrea. New algorithms for the top-k planning problem. In Proceedings of the scheduling and planning applications workshop (spark) at the 24th inter- national conference on automated planning and scheduling (ICAPS), pages 10-16, 2014.
[38] T. Ren, G. Chalvatzaki, and J. Peters. Extended tree search for robot task and motion planning. arXiv preprint arXiv:2 2021.
[39] R. Chitnis, D. Hadfield-Menell, A. Gupta, S. Srivastava, E. Groshev, C. Lin, and P. Abbeel. Guided search for task and motion plans using learned heuristics. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 447-454, IEEE, 2016.
[40] S. Niekum, S. Osentoski, G. Konidaris, and G. Barto. Learning and generalization of complex tasks from unstructured demonstrations. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5239-5246, 2012.
[41] T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanzhez-Gonzalez, E. Grefenstette, P. Kohli, and P. Battaglia. Compile: Compositional imitation learning and execution. In International Con- ference on Machine Learning (ICML), pages 4418-3428. PMLR, 2019.
[42] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations (ICLR), 2019.
[43] M. A. Toussaint, K. R. Allen, stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
Smith, and J. Tenenbaum. Differentiable physics and
[44] A. Arora, H. Fiorino, D. Pellier, M. Métivier, and S. Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33, 2018.
[45] Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022.
[46] M. Andrychowicz. F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[47] B. Da Silva, G. Konidaris, and A. Barto. Learning parameterized skills. arXiv preprint arXiv: 1206. 6398, 2012.
[48] J. Chang, N. Kumar, S. Hastings, A. Gokaslan, D. Romeres, D. Jha, D. Nikovski, G. Konidaris, and S. Tellex. Learning deep parameterized skills from demonstration for re-targetable visuo- 2019.
motor control. arXiv preprint
[49] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinchart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[50] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. OPAL: Offline primitive discov- ery for accelerating offline reinforcement learning. International Conference on Learning Representations (ICLR), 2021.

---

[51] L. Illanes, X. Yan, R. T. Icarte, and S. A. Mcllraith. Symbolic plans as high-level instructions for reinforcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling (ICAPS), volume 30, pages 540-550, 2020.
[52] V. Sarathy, D. Kasenberg, S. Goel, J. Sinapov, and M. Scheutz. SPOTTER: Extending sym- bolic planning operators through targeted reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1118-1126, 2021.
[53] A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. International Conference on Machine Learning (ICML), 2001.
[54] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. In International Conference Learning Representations (ICLR), 2019.
[55] G. Konidaris and A. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems (NeurIPS), 2009.
[56] E. Brunskill and L. Li. Pac-inspired option discovery in lifelong reinforcement learning. In International Conference on Machine Learning (ICML), pages 316-324, 2014.
[57] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[58] K. Gregor, D. J. Rezende, and D. Wierstra, Variational intrinsic control. arXiv preprint arXiv:1611.07555 2016.
[59] V. Thomas, J. Pondard, E. Bengio, M. Sarfati, P. Beaudoin, M.-J. Meurs, J. Pineau, D. Precup, and Y. Bengio. Independently controllable features. arXiv preprint 100.00/280000 2017.
[60] K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman. Meta learning shared hierarchies. International Conference on Learning Representations (ICLR), 2018.
[61] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations (ICLR), 2019.
[62] M. Riemer, M. Liu, and G. Tesauro. Learning abstract options. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2018
[63] P. Ranchod, B. Rosman, and G. Konidaris. Nonparametric bayesian reward segmentation for skill discovery using inverse reinforcement learning. IROS, pages 471-477. IEEE, 2015.
[64] C. Daniel, H. Van Hoof, J. Peters, and G. Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3):337-357 2016.
[65] R. Fox, S. Krishnan, I. Stoica, and K. Goldberg. Multi-level discovery of deep options. In CoRL, 2017.
[66] A. Mandlekar, D. Xu, Martin-Martin, S. Savarese, and L. Fei-Fei. GTI: learning to general- ize across long-horizon tasks from human demonstrations. In Robotics: Science and Systems XVI, 2020.
[67] D. Tanneberg, K. Ploeger, E. Rueckert, and J. Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 47004499 2021.
[68] P. Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:21 10.001177 2021.
[69] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. International Con- ference on Machine Learning (ICML), 2017.
12

---

[70] B. C. Da Silva, G. Baldassarre, G. Konidaris, and A. Barto. Learning parameterized motor skills on a humanoid robot. In 2014 IEEE International Conference on Roboties and Automa- lion (ICRA), pages 5239-5244, IEEE, 2014.
[71] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprini urXiv:2010.11944, 2020.
[72] M. Grounds and D. Kudenko. Combining reinforcement learning with symbolic planning. In Adaptive Agents and Multi-Agent Systems (AAMAS)), pages 75-86. Springer, 2005.
[73] F. Yang. D. Lyu, B. Liu, and S. Gustafson. PEORL: Integrating symbolic planning and hier- archical reinforcement learning for robust decision-making. arXiv preprint arXivv1804.077 2018.
[74] M. Jin, Z. Ma, K. Jin, H. H. Zhuo C. Chen, and C. Yu. Creativity of Al: Automatic symbolic option discovery for facilitating deep reinforcement learning, arXiv preprint 122.:221...0936.. 2021.
[75] Achterhold, M. Krimmel, and J. Stueckler. Learning temporally extended skills continu- ous domains as symbolic actions for planning, arXiv preprint arXiv:2207.05018, 2022.
[76] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Ser- manet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:220.001691 2022.
[77] S. Li, X. Puig, Y. Du C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch. Pre- trained language models for interactive decision-makee maasin.. arXiv preprint arXiv:2202.01771, 2022.
[78] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Ex- tracting actionable knowledge for embodied agents. arXiv preprini
2022.
[79] H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling, Learning symbolic models of stochastic domains. Journal of Artificial Intelligence Research, 29:309-352, 2007.
[80] H. H. Zhuo, Q Yang, D. H. Hu, and I.. Li. Learning complex action models with quantifiers and logical implications. Artificial Intelligence, 174(18):1500-566) 2010.
[81] Q Yang, K. Wu, and Y. Jiang. Learning action models from plan examples using weighted max-sat. Artificial Intelligence, 17102-3):077-143, 2007.
[82] H. H. Zhuo, T. Nguyen, and S. Kambhampati. Refining incomplete planning domain models through plan traces. In International Joint Conference Artificial Intelligence (IJCAI), 2013.
[83] P. Verma, S. R. Marpally, and S. Srivastava. Discovering user-interpretable capabilities of black-box planning agents. In International Conference on Principles of Knowledge Repre- sentation and Reasoning (KR), 2022.
[84] A. Mandalika, S. Choudhury, O. Salzman, and S. Srinivasa. Generalized lazy search for robot motion planning: Interleaving search and edge evaluation via event-based toggles. In Pro- ceedings of the International Conference on Automated Planning and Scheduling, volume 29, pages 745-753, 2019.
[85] R. Chitnis, L. P. Kaelbling, and T. Lozano-Pérez. Learning quickly to plan quickly using modular meta-learning. 2019 International Conference on Robotics and Automation (ICRA), pages -771 IEEE, 2019.
[86] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. The International Journal of Robotics Research, (M66-7):666-84. 2021.
13

---

[87] J. Ortiz-Haro, J.-S. Ha, D. Driess, and M. Toussaint. Structured deep generative models for sampling on constraint manifolds in sequential manipulation. In Conference Robot Learn- ing, pages 213-223. PMIR, 2022.
[88] E. Ugur and J. Piater. Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 2627-2633. IEEE, 2015.
[89] B. Ames, A. Thackston, and G. Konidaris. Learning symbolic representations for planning with parameterized skills. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 526-533. IEEE, 2018.
[90] Loula, Silver, K. R. Allen, and Tenenbaum. Discovering symbolic planning language from continuous experience. In CogSci, page 2193, 2019.
[91] S. James, B. Rosman, and G. Konidaris. Learning portable representations for high-level plan- ning. In International Conference on Machine Learning (ICML), pages 4682 1691. PMLR, 2020.
[92] A. Curtis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez and L. P. Kaclbling. Discovering state and action abstractions for generalized task and motion planning. In Proceedings of the AAAI Conference Artificial Intelligence, 2022.
14


Reviews:
[{"review_id": "lotVFWsB9f0C", "reviewer": "Reviewer_MQ6V", "summary": "This paper is about learning skills in task and motion planning problems via demonstrations. Learning a skill involves learning a goal-conditional policy together with a sub-goal sampler, as well as an operator that defines the logical state transition (postconditions and effects) that the skill will induce, assuming some preconditions are true when the skill is invoked by the task and motion planner. \n\nThe main contribution of the paper is showing how to learn policies, samplers, and operators from demonstration data that includes: the task definition (including a set of objects relevant for that task), the continuous state sequence, and the continuous controls. The continuous state transitions are segmented at contact points and thus split into skill dataset segments. These segments are lifted so that they are not grounded to particular objects, but types of objects. This is useful for learning general logical operators.       ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is comprehensive and generally well-written. Like many other related papers in this area, this paper loses some readability because of the heavy use of notation and definitions that are used to explain symbolic planning. This is unfortunately very hard to avoid, but I think the reader would benefit from a reduction of formal notation in the writing, if it is done in a way that increases intuition about the definitions. The use of a running example could help with this. Aside from writing, I would like to mention the following strengths (S) and weaknesses (W):\n\nS1: The generalization results in the low-training data regime (a few hundred demonstrated trajectories) are interesting, especially as the number of object varies at test time.\n\nS2: Long-horizon planning with learned skills seems like it works very well for TAMP, and this could definitely inform future design of such planners.\n\nS3: Segmenting demonstration trajectories at contacts makes a lot of sense for pick-and-place tasks.  \n\nW1: Lifting of demonstration trajectories that act on specific objects to more general types of objects seems error-prone, and I would have loved to see a more thorough analysis about how many logical state transition errors are introduced in the demonstration data via this pre-processing step. It seems brittle.\n\nW2: The tasks selected for evaluation seem a bit simple, with only a few objects at training time (1-3) for many environments and (1-2) more at test time. I would loved to have seen more complex sequences of sub-goals involving more objects in the training environments.                  \n\n          ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation for a weak accept is based on the strengths of the paper and particularly S1, namely generalization of learned skills, including goal-conditioned policies and logical transition operators, with a few hundred demonstrations. It's an interesting result.   ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The issues that I would love to see fixed in the revision would be W1 and W2. They would both make the paper's analysis and conclusions more robust, and more convincing in terms of the reliability of the skill learning method.  "}}, {"review_id": "GKjsPNokkN", "reviewer": "Reviewer_P5b1", "summary": "This work proposes a method for learning sets of pddl-style operators along with their associated low-level control policies from demonstrations. These \"neuro-symbolic skills\" can then be used in a straightforward bi-level search to solve challenging task and motion planning problems. The approach assumes knowledge of a set of predicates which are sufficient to define an abstraction of the low-level state, as well as a deterministic dynamics model for the environment. However, where traditional TAMP approaches also require hand-engineered operators, and associated control policies, this work shows how these can be learned directly from the given demonstrations. Experiments show that this approach can be data efficient, and produce skills which can generalize to new test problems, containing unseen numbers/placements of objects.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The idea of learning operators which can be used with AI planning techniques *and* associated low-level control policies is promising. The approach can plan very long sequences of actions to achieve temporally extended goals.\n* I appreciate the similarity of the approach to existing sampling-based TAMP methods.\n* The domain-agnostic treatment facilitated by PDDL and object-centric state makes it easy to see how this method could be applied to new domains and tasks.\n* In general, I appreciate the clarity with which the approach is presented. And in particular, the thorough analysis of experimental results. The many ablations that the authors included helped shed light on the impact of the different aspects of the approach, as well as the difficulties inherent in each evaluation domain.\n\nWeaknesses:\n* I think that the idea that certain skills might be easier to describe in terms of symbols (as opposed to a low-level policy/controller) is interesting. However, I felt that the experiments didn't showcase this, since every demonstration is generated by a hand-engineered policy. It would really help sell the point if there was an experiment where the skill was more difficult to engineer, and demonstrations were provided by humans.\n* A related point is that all the of the training data are generated by systems that themselves use the same set of predicates and bi-level search. This seems a strong prior, and makes for very straightforward segmentation of training demonstrations. And the segmentation appears to be crucial for the success of the overall approach. It would be more convincing if the demos came from a human or some other system that has no knowledge of those predicates.\n* The assumption of fully-observable state is very strong, but I understand the problem is already hard enough, and hope to see this method extended to the more challenging and realistic partial observability. \n* I think the choice of top-k planning seems like a weakness since the high level search for action skeletons isn't influenced by failures in the low-level policy. These failures should ideally help prune large portions of the high level search space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the idea of using symbols as a scaffold for skill learning is interesting, and I like the fact that the learned operators allow these skills to be used by a standard PDDL planner to solve long-horizon tasks. The methodology described in this work is simple and clear, and the results are promising. Ideally, I would like to see this method used to learn from human demonstrations, but this is a good first step. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* I would like to see the performance of the learned operators compared to the hand engineered ones. I am curious whether the learned samplers would be more sample-efficient, given that they are trained on successful demonstrations."}}, {"review_id": "CS542dojRSR", "reviewer": "Reviewer_ZRBu", "summary": "This paper exploits pre-defined symbols to discover skills that enable symbolic and low-level plan generation. In an environment with continuous states and actions, assuming pre-defined symbols, symbol mappings, predicates, object ids, transition functions, and unsegmented demonstrations, the proposed system discovers skills that encode symbolic operators (precondition and effect predicates) and skill policies.  Importantly, the skills that encode <precondition, effect> symbols can be achieved via several low-level goals, which brings flexibility in planning: The system can produce multiple symbolic plans in the outer loop and checks out with which detailed policy the symbolic plan is possible in the inner loop. The method is sound, verified in a number of tasks, and compared against baselines. The problem is an important problem, assuming pre-defined symbols and building up everything on top of the pre-defined symbols bring limitations to the scalability of the method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is very challenging, important (and timely) to learn skills that allow robots to make both high-level (symbolic) and low-level reasoning from the continuous sensorimotor experience of the robot. High-level reasoning requires symbols and operators to be learned, and low-level reasoning and execution require learning of detailed flexible policies that are linked with the operator definitions. The authors well-address the latter topic, and in this sense, their contribution is important and significant. However, they need to clarify their contribution with respect to [11] and [12] as detailed below.\n\nSaying this, the differences between the current work and the previous work [11] and [12] are not properly provided. The main contribution is stated as learning policies compared to [11] & [12] where the parametrized policies were pre-defined in the previous work. Stating this in only one line and then describing the Neuro-Symbolic Skills (Section 3) and Bilevel Planning (Section 4) without much referring to it is misleading about the contribution. In [12], Chitnis et al. also partitions skill data, unifies transitions based on symbolic effects, find the preconditions and effects of operators, and learn action sampler. This roughly corresponds to 5.1 and operator learning and sampler learning parts of 5.2 in the current paper. I might be missing the details, however, the only difference of the current paper seems to be in step 5.2 - policy learning; whereas all other sub-parts were realized in some form in [11] and [12].\n\nTherefore, the authors need to explicitly and carefully emphasize the exact contribution at the algorithmic level and at the realization/implementation level. They also need to provide why [11] or [12] were not included in the experiments as baselines. Maybe they might not be directly applicable for different reasons, then these reasons should be clearly pointed out. Additionally, was it not possible to apply [11] and [12] in this problem setting with small modifications? This also needs to be explained.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposed, implemented and verified a sound method for a challenging and important problem. However, it looks incremental in its current form.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please clarify the detailed contribution with respect to [11] and [12] in algorithmic and implementation levels.\n-  The writing of the paper is clear. However, I am not sure if it requires such extensive and a bit complicated notation. Is it possible to simplify to improve the accessibility to a wider audience?\n- The description of KD2 can be improved. It is not clear whether creating multiple symbolic plans or the flexibility in the inner loop is more important.\n- Please clarify whether object types were predefined or automatically generated from the beginning and also in 5.1 Lifting subsection.\n- In Policy Learning, please provide more details (possibly in the Appendix). Is only u_i learned/predicted? How about u_{i+1:k}, i.e. the parameters of the action execution? Why/how a parameter from a single (arbitrary) time-step represents the policy? Please clarify.\n- Is it possible to learn a skill with pre-conditions and effects that include a variable number of objects? As far as I see it is not possible. Is it an important limitation? How to extend the method to address this?\n- In the experiments, what happens if the training tasks include 3-4 buttons/cups and test tasks include 2-3 buttons? As far as I see, all 3 object types are required to be included in the preconditions (as they are all common in all training data), and therefore the method will fail with 1-2 objects. Please clarify whether this is the case, discuss whether this is an assumption, and re-discuss the generalization capability of the method taking into account this.\n- Achterhold et al. (2022) also learn symbolic operators along and continuous control, but with an RL agent. They also do hierarchical planning. This work can be included in related work or discussion\n\nAchterhold, Jan, Markus Krimmel, and Joerg Stueckler. \"Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning.\" arXiv preprint arXiv:2207.05018 (2022).\n"}}, {"review_id": "CRCVdKLE8Ei", "reviewer": "Reviewer_Y3Ww", "summary": "The proposed method tackles the TAMP abstraction problem in learning from demonstration by starting from a manually defined set of symbolic predicates and learning a parameterized policy in combination with symbolic operators and low level subgoal (operator parameter) samplers. There are simulated results.\n\n\"We learn skills that achieve certain symbolic effects (e.g., Grasped(robot, stick))\nfrom states where certain symbolic preconditions hold (e.g., HandEmpty(robot)).\"\n\nHuman programmers provide\n- a definition of what a low level state is\n- a definition of what low level actions are\n- the low level dynamics function\n- demonstrations in terms of low level states and actions\n- an ontology: values, features, and objects have types [This may be what breaks in the real world, but works in games.]\n- predicates which retrieve the values of features. The values of all possible predicates are known. In some cases the predicates seem carefully chosen to make the task work.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is on arxiv, the identity of the authors is public information, and this review was done knowing who the authors are. https://arxiv.org/abs/2206.10680\n\nSTRENGTHS\n\nI agree with the general philosophy of this paper\nthat reasoning, planning, and learning at the symbolic level as\nwell as continuous levels is useful. \n\nGood introduction.\n\nKD2: I agree considering alternative \"strategies\" or abstraction sequences\nis an important research area.\n\nCONCERNS (some philosophical)\n\nI disagree with the philosophy\nthat abstractions should be correct representations of the underlying problem\nand should be taken seriously. I prefer approaches where abstract plans are often not executable but\nsuggest where to search in lower level planning spaces.\nIn current robot dynamic legged locomotion, hierarchical control is used\nand more abstract representations are typically simplified (kinematic, dynamic, actuator, ...) \nmodels that approximate\nthe lower levels but the output of the simplified models cannot be exactly obeyed by the lower\nlevels. The simplified models \"suggest\" where the lower level planners should search.\n\nI would argue that the failure of early symbolic AI approaches to\nrobotics had to do with \nneeding the abstract plan to be correctly executable.\nThe proposed sampling-based approach only\nsamples low-level plans that conform to the abstract plan. Low level\nplanning approaches that explore a region of the low level planning space\nsuggested but not limited by a high level plan may be more useful.\nSimilarly, using imperfect or loose analogies may be a more effective way\nof making use of abstractions than treating an abstraction as a \"correct\"\nrepresentation of the original problem. Abstractions where abstract plans work in the abstract\nbut don't always work or are not executable in the real world, and vice versa, may be the norm.\nThe role of abstraction may be to merely suggest good\nstarting points or search directions for low level search.\n\nAnother issue with this paper is the manual feature selection.\nAs Patrick Winston often said, if you have the right features, the\nproblem is easy, and practically any method will work. If you don't\nhave the right features, then no method really works well.\nI am concerned that this work is doomed to succeed, with small numbers\nof carefully chosen relevant features. For example, CupFilled is created\nand MADE A CONTACT EVENT to get the approach to work on pouring.\nThis trivializes that task.\nOur experience with good old fashioned AI (GOFAI) suggests that\nit has been difficult to find the implementable predicates (symbols) for\nreal world robotics in the past. Has anything changed?\n\nCould the paper discuss when useful abstractions are likely to exist and be findable, before a problem is solved? Note that after a problem is solved it is often easier to find useful abstractions to solve it.\nIs it true that a successful abstract plan (as in line 132) always exists\nwhen a successful low level policy exists? This is not clear to me.\nA related question is whether a search in abstract space is guaranteed\nto find a successful low level policy that does exist.\nWhat properties of abstractions must hold for this to be true?\nIt may be that the only abstraction that works is no abstraction at all,\nS==X. Consider a parity function, for example.\n\nI am concerned the evaluation tasks as implemented in the simulator\nare toy problems.\nGripping objects and sticks, door/door-handle manipulation, using the stick as a tool, and pouring are all trivialized. Small numbers of relevant predicates are available for each task. Do sticks bend or break? Do stick tips slip? Does liquid splash during pouring? These are not\nsimple skills that we can provide low level policies for that work well in general. As mentioned before,\nCupFilled is made a contact event to make pouring work. A simple change to the work is to make all predicates from all task available to each task.\nLots of irrelevant predicates should cause segmentation and partitioning to fail.\n\nThe same argument goes for object types and objects. A relatively small number of relevant object types and objects are defined for each task. Are furniture objects? Lights and light switches? Interior decorations like pictures and rugs? How about the floor, wall, and ceiling? How about dust bunnies?\nHow are the objects involved in a task known? Are they all the objects that were present in the corresponding demonstration? Or just the ones changed (which ignores useful objects like tables)?\n\nI note that demonstrations are generated with handcrafted\nenvironment-specific policies. If doing the demonstrations is a solved\nproblem, why are we bothering to learn to do these tasks?\n\nAll the tests/experiments are done in simulation. What issues\nexist concerning the sim-to-real gap when applying these plans and skills on a\nreal robot?\nWill this approach work in the real world? Will it\nrun out of gas, as previous symbolic approaches to robotics have.\nIs this work doomed to succeed because of the tasks and environments chosen to work with and the use of \"clean\" simulations? Simulated and \"symbolic\" (often man-made) environments and systems\n(such as the ones used as tasks in this work) easily lend themselves to symbolic representations. Messy real world and \"natural\" systems don't.\n\nI note that object types seem to be pre-defined rather than learned.\nIt is known what a button is, and that it can be pressed. This is a second manual intervention, in addition to defining predicates, whose implications are not discussed in the paper.\nThe robot\ncannot construct a new object and discover it is a button.\nAnalogies must be exact, rather than loose or suggestive.\nThis makes lifting easy and well defined. How is the case handled where an\nobject has properties like a button, but is not completely like\na button in all ways, or fit the type definition?\n\n\"If sampling fails for an operator sequence, the outer loop continues to another sequence, addressing KD2.\"\nMuch more information needs to be provided about when the system gives up on\ntrying to attain a particular subgoal. KD2 should be modified to say \"an agent\nshould be able to EFFICIENTLY consider multiple skill sequences ...\".\nConsidering alternative \"strategies\" or abstraction sequences is not all that useful if it is not efficient.\nThe planning algorithm proposed in section 4 is likely to be very inefficient.\n\nA concern is the notion that there are discrete objects. The notion of object\nas presented in the paper breaks down with liquids, granular materials,\nand deformable materials like peanut butter and clay (plastic materials?).\nIt also does not handle materials that are consumed, created, or transformed during use\n(food, fuel, cooking, ...) well. It is difficult to reason about object construction and\nstructural failure (fracture, breakage, wear, fatigue, ...).\nWhat does it mean for an object to be wet? Glued or nailed to another\nobject?\n\nThis paper reminds me of early work in machine learning on\nsymbolic concept learning, such as https://axon.cs.byu.edu/~martinez/classes/778/Papers/Mitchell.ch2.pdf\nThat work was useful, but eventually superseded with statistical or\nnumerical methods. Will the same happen to this work?\n\nI agree that segmenting trajectories at contact state changes makes sense.\nHowever, it probably makes sense to have a hierarchy of segments.\nAt the lowest level one has segments A, B, C ...\nThe next level might have AB, BC, ...\nThe next level might have ABC, ...\nThis helps deal with the case that multiple contact changes lead to a\nknown state (ball rolls down an inclined plane into a V shaped corner or\nonto a floor).\nIt always ends up at the bottom of the V or on the floor,\nbut the contact sequences may vary.\n\nThe learned subgoal sampler as designed is unlikely to explore out-of-distribution parameters. For example let's say it needs to rotate a bolt or screw in a screw, and in the evaluation/real-world scenario, it happens that it needs to exert way more force than it is used to, simply because there might be more friction or corrosion, how likely is it to apply out-of-ordinary/distribution force? It cannot reason about the possible reasons for the screw being stuck.\n\nHow is the \"true\" goal of a demonstration known? How are irrelevant actions ignored? For example, human breathing is usually ignored in demonstrations, but this is not obvious without strong priors.\nHow are changes in the environment not caused by the demonstrator ignored (water evaporates or leaks,\nobjects may move on their own or due to gravity, ...)?\nLots of irrelevant features (predicates) should break this ability to tell what is relevant in a demonstration.\n\nAdd and delete operators were designed for features that had true or false values, and does not seem like the most useful semantics for robots.\nI don't see how add and delete operators handle cases where a value is incremented in the paragraph on operator learning starting on line 184.\nFor a feature x with a value 3.2, to add 1.9 to x one doesn't delete x and then add x with the new value.\nWe don't want x in E- and in E+.\nIt makes more sense to have a set M of features whose values are modified.\nWhat is the semantics of <e+,e-> = <s'\\s,s\\s'> when values of features are modified?\n\nSUGGESTIONS\n\nIt is much easier to read this paper AFTER reading the supplementary material.\nIt would be nice if that were not the case. Many comments below (written as I\nread the paper first) ask for definitions and examples, which are provided in the \nsupplementary material. The paper does not adequately stand alone.\nThe examples of symbolic\nplans and how the underlying primitives work in the supplement are really helpful. It would be great if one of them could be in the paper itself.\n\nI don't understand why the term \"neural\" is used in this paper.\nThere is no connection to biological neurons. The neural sub-goal conditioned\npolicy can be represented by any parameterized function framework,\nas line 205 points out.\nA subgoal sampler is just something that proposes instances from\na density function, which again can be represented in many ways, either\nwith a parametric function approximator or a nonparametric representation\nwhich retains all the instances in the training set and picks one or\nlocally interpolates to propose a candidate.\nIs the word neural used because that is the current fad in function\napproximation? Is there any evidence that the neural network architectures\nused in this paper are superior to other function approximators for the\npurposes they are used in this paper? There are only a relatively small\nnumber of training examples (less than 10,000), so the argument for\ndeep neural networks does not really apply.\nI would call what is proposed a hybrid-symbolic-and-numeric algorithm, or a hierarchical algorithm.\n\nSections 3, 4, and 5 would be easier to read if a simple example was used\nthroughout. Maybe navigation through a series of rooms, where each room\nwas an abstract state, and skills existed to find doors, open them, and\nmove to another room.\n\nIt might be useful to note that KD1 implies some lookahead so later (soft and hard) constraints are\ntaken into account. This is what trajectory optimization and backpropagation\nin time do.\n\nI can't figure out what the tasks are just from Figure 1.\n\nAn environment (vs. a task) has object types (button), but not specific objects\n(button1, button2, ...)?\nOr did you want to include the object space O in the environment tuple\n< Alpha, O, U, f, Phi >?\nThe set of objects in a task should be a subset of the set of objects\nin the environment (irrelevant objects should be able to be present).\n\nIn top-k planning, how are the candidate plans made non-trivially\ndifferent? Such a planner could generate plans that have trivial\ndifferences.\n\nThe notation in line 165 may not be familiar to the reader.\nTo make your paper more accessible you should explain what <s' \\ s,s \\ s'>\nmeans in this context.\n\n165: Can a segment be mapped to more than one skill dataset?\n\nHumans learn with 1 or a few demonstrations.\nThis approach seems to need hundreds of demonstrations. Why?\n\nThe paper is good at presenting technical definitions and details. It would be even better if intuition was given, as well as definitions in words.\n\nWords like operator, ground atom, and lifted atom seem to be necessary to understanding sections 3-5, but aren't explained until section 5.\n\nHow is this different from various other hierarchical approaches in\nmachine learning that simultaneously learn parameterized skills and/or\nhow to sequence them? Perhaps an explanation of how the components of\nthe AI planner relate to the components present in many other\nhierarchical planning approaches would be beneficial.\n\nWhat elements differentiate an AI planner from other hierarchical\napproaches that, at a high-level, also plan in terms of discrete,\nabstract skills? Intuitively, and in layman's terms, what do each of\nthe referenced components (operators, ground atoms, lifted atoms, add\neffects, delete effects, etc.) of the AI planner do and how do they\nfit into the system as a whole? Elaborating on these points prior to\nand/or within section 2 would make it much easier to follow what all\nthe symbols mean and remember their significance throughout the rest\nof the paper.\n\nThe term \"operator\" is not described in layman's terms.\n\nLine 22: \"We are motivated by cases where it is easier to design [] or learn [] symbols than it is\nto design skills.\"  What are examples of these cases? It would be\nhelpful if the authors could elaborate on why it may be easier to\nlearn/design symbols than it is to design skills. For humans, many of\nus can learn skills (riding a bicycle, throwing a pot on a potter's\nwheel) for which we cannot easily articulate predicates (features we are\npaying attention to) or how we do the task. Most bicycle riders are\nunaware of the non-minimum phase nature of bicycle dynamics and\nthat you initially turn the front wheel in the opposite way you wish to\nultimately turn.\n\nLine 112: What are the limits of add/delete effects? Do they limit\nwhat can be done with an attribute and its value in any way?\n\nLine 172 - 182: The concept of \"lifting\" is described in detail for\nthe first time after already being referenced multiple times.\n\nLine 322: \"To the best of our knowledge, our work is the first to\nlearn operators, samplers, and policies that are compatible with\nsearch-then-sample TAMP [29, 30, 31].\"  What makes the learned\noperators, samplers, and policies compatible with search-then-sample\nTAMP; and why has this not been the case in other works? To make this\nclaim, it should be clear what compatibility means and how it is\nmeasured.\n\nThe result on the doors environment suggests that success it not possible without\nthe manually added motion planning skill. An ability to recognize where a planning\nskill is needed, and even an ability to default to a explicit planner in this case would\nstrengthen the result.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Ignoring philosophical differences with\tthe approach, I\tthink my\nconcerns focus on whether the evaluation is doomed to succeed.\nThe predicates are carefully chosen. Additional\tsymbolic\ninformation is provided\tby defining object (and value) types.\nNot a lot of irrelevant\tinformation has\tbeen provided.\nThe simulations\thave been greatly simplified. In this\nkind of\twork the scale or numbers of X (X = predicates,\tobjects, tasks,\ndemonstrations,\t...) has to be large for the work to be\nconvincing.\nThe planning algorithm presented is clearly inefficient\tand will\nnot scale well.\n\nPOST REBUTTAL COMMENT\n\nI am still in favor of accepting this paper.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Things that can be fixed in a reasonable amount of time:\n\nWRITING\n- improve and clarify writing (see my comments on that)\n- make the paper stand alone more effectively, by moving some definitions and examples from the supplementary materials to the paper.\n\nIN SIMULATION\n- add large numbers of irrelevant predicates and operators\n- see what happens if you don't filter out rarely used operators\n- see what happens if you don't reject out of bounds subgoals\n- wider variety of simulated tasks - ideally non-symbolic tasks and not effectively pick and place: paint something? repair something with glue, tape, or string? shape something such as clay or dough?\n\nEFFICIENT PERFORMANCE\n- explore several abstract plans in parallel rather than serially. This would deal with the issue of wasting a lot of time exhaustively sampling bad plans, and still allow adequate sampling on other plans.\n\nEXTEND APPROACH\n- start with demonstrations alone, measured with low level sensors (not manually defined high level\nperceptions or predicates) as suggested in C.4.\n\nBigger changes I don't expect to be made, but could be made\n- real robot implementation\n\nMORE COMMENTS ON WRITING (to get around stupid character limits in the web interface):\n\nOverloading lambda for object type and real valued features is confusing\n\nIn partitioning grouping segments if their affects are the same “up to object\nsubstitution” is unclear. Does this mean any object of any type can be substituted into a\npredicate\nor does it mean only different object instances within the same type can be\nsubstituted.\n"}}]

Author rebuttal:

Decision: Accept (poster)