Learning Reusable Manipulation Strategies
Authors: Jiayuan Mao
Tomás Lozano-Pérez
Joshua B. Tenenbaum
Leslie Pack Kaelbling
Venue: CoRL 2023 Poster (2023)
Keywords: Contact Modeling and Manipulation
Task and Motion Planning

Abstract: Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks." Even from a single demonstration, such as using soup ladles to reach for distant objects, we can apply this skill to new scenarios involving different object positions, sizes, and categories (e.g., forks and hammers). Additionally, we can flexibly combine various skills to devise long-term plans. In this paper, we present a framework that enables machines to acquire such manipulation skills, referred to as "mechanisms,'' through a single demonstration and self-play. Our key insight lies in interpreting each demonstration as a sequence of changes in robot-object and object-object contact modes, which provides a scaffold for learning detailed samplers for continuous parameters. These learned mechanisms and samplers can be seamlessly integrated into standard task and motion planners, enabling their compositional use.

Learning Reusable Manipulation Strategies
Jiayuan Mao Joshua B. Tenenbaum Tomás Lozano-Pérez Leslie Pack Kaelbling Massachusetts Institute of Technology
Abstract: Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks." Even from single demonstration, such as using soup ladles reach for distant objects, can apply this skill to new scenarios involving different object positions, sizes, and categories (e.g.. forks and hammers). Addi- tionally, we can flexibly combine various skills to devise long-term plans. In this paper, we present a framework that enables machines to acquire such manipulation skills, referred to as "mechanisms;" through a single demonstration and self-play. Our key insight lies in interpreting each demonstration in robot-object and object-object contact modes, which provides scaffold for learning detailed samplers for continuous parameters. These learned mechanisms and samplers can be seamlessly integrated into standard task and motion planners, enabling their compositional use.
sequence of changes
Keywords: Contact Modeling and Manipulation, Task and Motion Planning
1 Introduction
Humans possess an exceptional ability to acquire and generalize manipulation "strategies." Even from a single demonstration of using a soup ladle to reach a distant object (Fig. la), we can generalize and reuse this strategy in various novel scenarios: to different object positions and sizes, and even to diverse object categories like forks and hammers. Furthermore, humans can combine these strategies devise long-term plans, perhaps using soup ladle to hook an apple, place it into bag, and move the bag to shelf, dramatically expanding the scope of manipulation abilities.
salient feature of these manipulation tactics is that they can be expressed as a sequence of basis manipulation operations, characterized by varying contact modes between robots and objects. For example, as illustrated in Fig. 1 and c, the "hook-using" strategy comprises a series of four contact modes: the free movement of the arm, tool grasping while applying contact force between the tool and the target, tool placement, and ultimately, target grasping. The continuous parameters of these operations, in principle, can be produced by generic samplers and motion planners, but planning in terms of these generic basis operations can be very slow due to long planning horizon with substantial branching due to choice of basis operations and continuous parameters.
To tackle these challenges, this paper presents a framework that equips machines with the ability to learn, generalize, and reuse such manipulation strategies. referred as "mechanisms." through single demonstration and subsequent self- -elf-play in distribution of target problems. The key insight driving our framework is the characterization of each mechanism as sequence of contact mode changes between the robot and objects, complemented by specialized sampler that generates grasps, contacts, and trajectories tailored specifically for the mechanism. Our framework takes an explanation- based learning approach [1, 2]. departing from conventional methods that learn policies or parameterized trajectories from large numbers of demonstrations. particular, our framework extracts an abstract representation that explains the underlying contact interactions between objects from the demonstration, then during the self-play stage of mechanism learning, the agent explores feasible actions that align with the demonstrated contact mode sequence, generalizing to different objects and initial configurations. Leveraging successful trials from self-play, we train samplers
Project page: http://cocccpttaaaaccoo//mmccmae
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Suppect
Support
Inidial State
Hand
Ladle
Spuan
Flour
Hand
Grasge
Grasp Ladle
Hook Spoon
Hand
Grasp
Tadik
Spoon
Support
Floor
Comtact: (iv.n)
Place Ladle
Hand
Support
Spoon
Support
Floor
Spoon
Grupp
Support: (p.n)
Grasp Speon
Hand
Spuon
Support
(a) Demonstration of using soup Indle to reuch for the spoon
(c) sequence of contact mode graphs the process objects, with continuous parameters. of using the soup ladle to reuch for the spoon.
(b) The contact mode graph between
Figure 1: A single demonstration can teach reusable multi-step manipulation strategy.
place(L.8.)
place(L.8.)
grasp(L, B.)
grasp(L
St
move(L, 02) grasp(L.
%
St
move(L, (2)
push 02)
pussA 8,)
grasp(L.. Ba)
moue(L B.)
%
TROPELL move(L, 8.)
push(L, 8-)
push(l, 0x)
(a) Search the large hybrid space of discrete mode families and continuous motion parameters.
(b) Search in the hybrid space guided by the mechamism. Only continuous purameters needs to be searched.
Figure 2: (a) Planning with hasis manipulation operations usually involves a search in the hybrid space of discrete contact mode families (free, holding, pushing. etc.) and continuous parameters (grasping pose, moving trajectories, etc.). (b) Learning mechanisms can help reduce the search space by specifying sequence of discrete transitions between contact mode families.
that are tuned specifically for each mechanism. The learned mechanisms and samplers can then be recombined to efficiently devise long-horizon plans for novel goals in novel environments, by reducing the effective search horizon and focusing the sampling process during planning.
This formulation introduces significant capacity for generalization via abstraction and compo- sitionality: by extracting the contact-mode sequence from the demonstration, retain its most causally important aspects while abstracting away many irrelevant details, and by representing learned mechanisms in a form similar to basis operations, we are able to leverage task-and- annd-ootoon planners to obtain a truly compositional system. The contributions of this paper are: a novel representation of complex manipulation actions terms of mechanisms, an algorithm for learning new mechanisms from a single demonstration and subsequent self-play, and planning framework for integrating new mechanisms with other manipulation primitives, including those that are briefly dynamic [3], to solve novel problems.
2 Planning with Contacts and Mechanisms
Our framework based on hybrid task and motion planning. We begin with small set of generic basis manipulation operations corresponding different contact mode families [4] between robot and objects (Table 1). Sequences of these basis operations form rich class of manipulation strategies, such as using soup ladles to reach for distant objects, or pushing plates to the table edge to pick them up. In theory, a complete search algorithm could discover such strategies and find sequences of them to solve difficult novel problems, without any demonstrations at all, as illustrated in Fig. 2a. However, planning at this level is inefficient to the point of infeasibility. Therefore, this paper, we will explore learning "macros" of basis operations, using a compatible representation that will allow learned mechanisms to, themselves, be composed solve harder problems.
2.1 Basic Domain Representation
We adopt representation similar those used in the task and motion planning (TAMP) literature [5]. Formally, given space S of world states, TAMP problem can be defined tuple (S, so G, A, Here, 8o S denotes the initial state, which includes the shapes and poses of all objects. G C

---

Name
Args.
Description
transit
(emply)
Move the robot without object collisions.
transit-cont
2x, 2s
Move without holding anything. but have contact with 2x. 2x is supported by Ps.
grasp
2x,
Grasp the object ?x that currently supported by ?s.
place
2x, ?s
Place object 2x onto ?s. ?x should be currently held by the robot.
move
2x
Move while holding ?x.
move-cont
Px, Py.. ?s
Move while holding ?x and ?x have contact with 2y. 2y is supported by ?s.
Table 1: The list of basis operations. Continuous parameters are omitted. "cont" refers to "contact."
(saction wove contact
(inechantse book
sparsnctors (?tool ites Itarget itex Psupport
(?tool iite Ptanget Stee 2support rode graph (handsfree) (sapport Ptool Psupport) (support Farrgt Psupport))
?paras -ooeaat---oe-ppaaa hqt trajectory) sprecondition (and
sprecondition is initial
(holding #tool)
scertified
23 goal. the mechantsn (suportt Ptool ?support)
(support Ptarget 2support)
(valld- coaldd-contact-enn -otoeatttt---v--areee
Pparan) 1: pointsenoreal
(holding Itarget))
sorlidd-ootatt-roe----e trajeetryyyyoory.. ?paran 2qt))
factions (andered i; sequence contact prinitives (grosp Ptool ?support)
reffect (and
(assign (robot-canfig) ...) update rebot position
(rove-cont Ptool ?target Psupport]
(assign (pose Htoo))
update tool position
(place Ptool 7support)
(assign (pose ?target)
target possiiion
(grasp Ptarget Psupport))
(a) The modeling of the robot basis operation move-with- contact using STRIPS-like syntax..
(b) The procondition (initial contact modes), goal (target contact modes), and the sequence of actions for a mochanism.
Figure 3: The syntax used to model basis operations and mechanisms.
represents the goal specification, often expressed as a logical expression evaluated based on a state (e.g.. holdinng(ppoon). A denotes a set of continuously parameterized actions that the agent can exccute, such as grasping and placing objects. Finally, T is a partial environmental transition model parameterized by two functions: the precondition
defined as T : :S x + S. Each action function , and the effect function eff.- The semantics of these functions is that for any state S and action E A, if pre, (s) holds, then T(s, a) eff, (s).
State representation. An environmental state is represented as tuple = (U,, P.). U, denotes a set of objects, including the robot and other physical objects, and we assume it is fixed during the execution of actions. Objects in Us will be referred to using names such as Soupl and Floor. The set P. contains state variables (atoms in STRIPS). Each state variable contains predicate name (e.g., pose), a list of object arguments (e-g., SoupLadle), and value (e-g.. the pose of the soup ladle in SE(3)). In addition to shapes and pose variables, P, also contains two sets of variables to represent the contact mode graph: holding( (xx) and support 2x, ?y), where 2x and ?y are instantiated with all objects in U, These variables describe whether the robot is holding an object ?x and whether the object ?x is supported by Zy, respectively".
Basis operators. The basis operators are specified parameterized operator schemas, (name, args, precond, effect, sampler). where name is the name of the schema, args is list of arguments, including both object arguments and continuous parameters. These continuous values can be generated by invoking the sampler, possibly conditioned on other aspects of the state, such as the shapes of the objects involved. The precondition precond and effect effect are logical expressions over variables in args and will be evaluated at the current state. A schema can be grounded into a concrete basis operation a by specifying its arguments. See Figure Fig. 3a for a concrete example and Appendix A.3 for more discussion.
2.2 Mechanisms
A manipulation mechanism is defined as a sequence of basis operations with a specialized sampler, in order to accelerate planning with generic basis operations. Formally, each mechanism is represented as a tuple of (args, precond, certified, actions, a sampler). args is a set of arguments. precond is the initial contact mode graph including holding and support relations. certified is the goal of the mechanism, which usually specifies the final contact mode graph. actions is an ordered list of primitive operations. sampler is the specialized sampler that can generate continuous parameters for all basis operations in actions. Fig. 3b illustrates the definition of the "hook-use" mechanism. In this case, three objects are involved: the tool object (e.g., the soup ladle), the target object (e.g., the
FFollowing the STRIPS convention, we will be using names such as ?x and 2y for variables and strings such SoupLadle for objects.
3

---

Crasp 0,)
(:nacro hook
sparameters (
Ptarget
Ptool Iten
conk/took 0g)
Psupport
scentified (
place(?tool 8y)
(holding Mtarget)
Goal: Block on the Skepe
Single Demo
Contact Modes and Goals
Self-Play
Learned Contact Distributions Compositional Planning
Figure 4: Learning process: extract contact modes and goals from demonstration; train specialized sampler via trial-and-error in new situations; add new mechanism planner.
spoon), and the support object (e.g., the floor). The goal of the mechanism is to grasp the target object that was initially out of reach. This macro contains sequence of four basis operations: grasping the tool from the support, moving while holding the tool and "indirectly" pushing the target object, placing the tool back the support, and finally grasping the target object. It also has an associated sampler that generates feasible grasps of the tool and contacts between the tool and the target object.
2.3 Planning with Basis Operators and Mechanisms
To plan using the basis operations, mechanisms, and samplers, employ simple bilevel search approach, similar to the adaptive algorithm in PDDL-Stream [6]. Illustrated in Algorithm 2 in Appendix A. we first apply symbolic STRIPS search algorithms with the fast-forward heuristic [7] to explore discrete action plans (i.e., candidate sequences of basis operations and objects to move). Subsequently, we use samplers to find suitable continuous parameters. We iteratively call the samplers associated with each operation to generate continuous parameters, such as grasps, contact surfaces, and trajectories. We simulate the grounded operators to verify their effects, and backtrack discrete plan if continuous parameters cannot be found. Mechanisms can be directly integrated into the bi-level search-based planner additional operators, with their specialized samplers. Intuitively, as illustrated in Fig. 2b, incorporating mechanisms in search prunes out the branching factor of possible contact modes and also improve efficiency in sampling continuous parameters.
new
The use of physical simulator verify action effects enables handle briefly-dynamic tasks, in which the robot controllers are position-based but may initiate situations in which the objects experience acceleration and velocity before they reach stable configuration. Examples include objects sliding down inclines, tipping upward when weights are placed on them (see Appendix A.2).
Learning New Mechanisms
Our goal is to learn mechanism from single demonstration and distribution of target problems. The demonstration includes sequence of robot actions and object contacts, as well as human-specified goal, which will be the certified effect in the new mechanism definition. We assume access to an environment simulator that can generate random initial configurations of objects such that the target mechanism is applicable. For example, in the hook-use case, the environment contains two objects of various categories placed on the table so that one object is within reach and the other object out of reach. The overall framework depicted in Fig. 4: The learning algorithm first extracts the contact-mode changes in the demonstration, resulting in sequence of basis operations. Next, it generates self-play trajectories that align with the basis-operation sequence but now with novel objects in novel initial configurations (e.g., using spoons reach for forks). This self-play step involves tridl-and-error interaction with the environment. Based on the successful trajectories, we learn sampler that generates mcchanism- speciifc contacts between objects (c.g., grasp tools in order to reach for distant objects), and finally add it to our repertoire of planning operators.
3.1 Extraction of Preconditions and Operation Sequence
We first segment the demonstration trajectory based on robot-object contacts: free, holding, and push motion. Each segment will correspond to step in the mechanism macro. Next, for each segment, we build contact graph between the robot hand, the object that is in contact with the robot (including

---

Algorithm Sampler Learning Algorithm
1: procedure SAMPLERLEARNINGienn m)
2: Initialize classifiers Sa and dataset D; for each continuous parameters in m.
3: for each iteration do
4: 80 env.reser()
b> Sample an initial configuration < RandomGrounding (m) >> Apply the mechanism on random set of objects 6: plan - CONTINUOUSSEARCHCC gm..gg...tn.... gm.certified, gm.actions)
5:
7:
for each continuous parameter do for all 0 sampled CONTINUOUSSEARCH for parameter do if f is in plan then Add (6, 1) to D;
8:
9:
b Successful samples Failed samples
10:
else Add (0, 0) to D; Update classifier fi using L and D;
11:
holding pushing, if any), and the object that is in contact with the held object ('indirect contact"), Finally, add all the objects that support the objects (exerting forces that point in the + direction). By completing the relationships among objects, we obtain the sequence of basis operations. The precondition of the mechanism operator corresponds to the initial contact modes. After extracting the initial contact modes and the basis operation sequence that are grounded on concrete objects in the demonstration (the soup ladle, the spoon, and the floor), we lift them into an abstract mechanism definition by replacing concrete object names with variables, as illustrated in Fig. 3b.
3.2 Sampler Learning
The sampler learning process is an iterative exploration within a simulated environment. Algorithm describes the high-level process of learning a sampler based on the success of generated samples in achieving the goal. In each iteration, we randomly sample initial configuration from the simulator and attempt to execute the mechanism objects present in the environment. Notably, instead of searching for plan to accomplish the mechanism's 's goal with all available basis operators, require the search algorithm to adhere to the sequence of basis operations derived from the demonstration. Specifically, we fix the discrete-level plan and employ generic samplers for grasping, contact, and trajectory generation to locate plans that satisfy the goal. All continuous parameters sampled during the search will be labeled as "I" or "0" based on whether they present the successful plan. Note that even if the environment does not contain distractors and the mechanism is always applicable, we still need to explore different choices of arguments (e.g., use which object reach for the other one).
Leveraging this labeled dataset of samples, we can train an additional sampler that generates mechanism-specific samples of continuous parameters. For example, in a scenario where soup ladle used to reach distant objects, it favorable to grasp the handle-a more brrchanism-ppeiiiee action, rather than generic grasps. This can be formalized learning a distribution of "successful" continuous parameters for actions within mechanism. We do this by training score function to rank samples produced by the generic samplers'. In essence, for each continuous parameter in the mechanism (grasping poses, contact surfaces, and trajectories), given dataset of samples and their success labels D (0, label), we train a classifier that estimates scalar value in [0, 1] representing the probability that can result in a successful application of the mechanism, using binary cross-entropy loss: During deployment, we use the generic sampler to generate a batch of samples, which we sort based on the predicted likelihood of success.
Note that although we train classifiers for individual parameters, these classifiers are designed to be conditioned on previously generated parameters as defined in the basis operation schema. Hence, their sequential application is capable of generating joint distribution of successful parameters for mechanism applications in an "auto-regressive" manner. The benefit of this factorized approach, as opposed to learning a single classifier over the concatenation of all parameters, its efficiency during the search process. If the execution of one step the mechanism fails, the search algorithm can immediately backtrack, thereby averting wasteful continuation into subsequent steps.
More sophisticated generative models will work, especially when the samples are hard to generate.
5

---

3.3 Implementation
Our algorithm for learning mechanisms is general purpose. In order to demonstrate its value, we have implemented a concrete system with the basis operators in Table 1.
Samplers for Basis Operators There are three types of continuous variables to be sampled for the basis operators described in Table grasping poses relative to an object (represented as SE(3) poses of the end-effector), placement poses (represented as SE(3) poses in the support object), contacts between two objects (represented as the SE(3) pose of object 1 in the frame of object 2), and robot arm trajectories (represented as a sequence of joint configurations). The samplers are designed to be generic, relying solely on geometry and not specific object semantics (e.g., soup ladle grasping). We include one here as an example and describe the full list in Appendix A.4.
Object Contact (C). For both robot-object and object-object contact, the object contact sampler, C(O,, O2 Toz, T,) takes five arguments, including the current holding object O₁ (or the robot gripper itself when not holding anything), the object to contact O₂ and its current pose Ta2, and the object that supports O2: O2 and its pose T.. It first randomly samples two surfaces, represented as a pair of (point, unit normal), (P1, and (p2, r2)) on O and O2 respectively. Since we do not consider pushing O2 "towards" the supporting object O,, we additionally require that n2 is perpendicular to ng, which is the direction of the support force from O2 to O2. Next, it solves for a transform T on On such that Tpt To2P2 and Tni TT2222 (essentially place pi on O at p2 and pointing towards 122 to excert force).
Learned Samplers Each mechanism has specialized samplers for all of its continuous parameter, organized into DAG, in which each parameter is sampled conditioned on its parents in the graph (a.k.a. previously sampled parameters). Each sampler has a scoring classifier that processes the target parameter along with any values it is conditioned on and predicts success likelihood in [o, 1]. We employ various encoders for different parameter types. For object shapes, we use PointNet++ encoder [8] process the point clouds. For poses, including 3D translation and 4D quaternion, utilize Multi-Layer Perceptrons (MLPs). For contact information, encode the contact points and normals using MIPs as well. Each of these encoders processes their corresponding inputs into a fixed-length embedding, and the embeddings are then concatenated into a single vector. Finally, we apply a linear transformation followed by a sigmoid activation to output the classification result. We train each classifier on 100 samples. For trajectory-typee to object-object contact trajectories encoded as the contact normal direction and the distance. We do not consider encoding and classifying the actual robot arm trajectory for grasping and placement motions, although it is, in principle, possible to encode them using appropriate encoders.
parameters, we only apply classifiers
4 Experiments
We conducted experiments in the PyBullet simulation environment, to evaluate the ability of our system to learn individual mechanisms and its ability reuse learned mechanisms in novel tasks. We describe real-world deployment of the system in Appendix B.3 and more videos on website.
4.1 Learning Mechanisms from Single Demonstrations
Setup. Our evaluation encompasses six distinct mechanisms, grouped into two categories: the first four tasks assess "tool-use," including (Edge) pushing objects to the edge of a table for pickup, (Hook) using tools to reach for distant objects, (Lever) flipping objects using heavy objects as levers, and (Poking) using tools poke objects out of a tunnel. The remaining two tasks fall under the "reasoning about stability" category, including (Center-of-Mass) achieving stable object placement on another object, Saape--nddBlockkey using objects blockers to prevent objects from falling off inclined surfaces (illustrated in Fig. 5). The object models are blocks, bricks, bowls, plates, documents, spoons, forks, soup ladles, hairbrushes, hammers, and calipers. The demonstrations are created by executing a manually written script in one specific initial configuration.
During training and testing, each method has access a distribution of initial configurations and goals. Each task consists of a randomly sampled initial configuration that includes target objects placed on the table and specific goal to be achieved (c.g., holding one of the target objects). We

---

Initial
Solution
Edge: hold(P)
Hook: hold(B)
Lever: hold(P)
Poke: hold(B)
CoM: suppon((), B)Slope: support(B, S)
Figure 5: Illustration of six mechanisms learned by our algorithm. Top row: initial configuration. Bottom row: solution found by sampling- campling-based planner. P: Plate. B: Block. S: Slope.
Method
Edge
Hook
Lever
Poking
CoM
Slope&Blocker
Basis Ops Only
89.4545.53
>600
523. 123.1829.22
>600
19.30=2.82
>600
Ours (Macro)
8.34
30.82.5.78
1.38:+0.31
500.00=50.01
1..58
148.57= 0.97
Ours Sample+Sampler)
0.57=10.05
1.84
1.55
97.76=10.66
4.11:0.94
Table 2: Average solution time and stderr of 10 trials, All methods have a 10 mins timeout.
design the distribution of initial object placements ensure the feasibility of the corresponding mechanism. For example, in the "hook-use" mechanism, we randomly place objects the table. Object 1, which is within the robot's reach, is selected from the following categories: soup ladle, hammer, spoon, and hairbrush. Object 2 is a box that is initially placed outside of the robot's reach. We provide more environmental details in Appendix B.
Results. Table summarizes the results of our experiments comparing our method to planning with the basis operators only, as well to ablation of our method which we learn the basis operation sequence for the mechanism but do not learn specialized samplers. Due the inclusion of novel object instances and categories in the test environments, simpler baselines, such as replay of the demonstration trajectory, have zero success rate, and are not included. The table shows the average time needed for each planning algorithm. Our method effectively learns mechanisms for all tasks. The macro-only version occasionally timed out, emphasizing the importance of both sequence and sampler learning. As an example, consider the Lever task, in which the robot needs to position an object on plate flip the plate up. Having prior knowledge of this strategy at the abstract basis operation level significantly simplifies the task. The learning of mechanism-specciff sampler, in this case, does not offer much additional benefit. On the contrary, task CoM presents a different scenario: here, the robot is tasked with learning a distribution for stably placing an object on a small block, which benefits greatly from learning a specialized sampler. The poking task difficult, even for learned samplers because involves a arm motion planning problem with a very small collision- -free region. We discuss more details on the importance sampler learning in Appendix B.2.
4.2 Planning with Learned Mechanisms
Setup. To test the compositionality of mechanisms, we evaluate different algorithms on two novel complex tasks, illustrated in Fig. 6. caliper, placed on the table to reach for a distant block. However, the caliper must first be pushed to the table side to enable successful grasp. Shook-then-Placcoo--Spppp the agent needs soup ladle as a tool to reach for distant object. Subsequently, the agent must use either the soup ladle or brown brick to act as a blocker, preventing the object from falling off slope. There is no additional training- we used mechanisms learned during the previous experiment. We design the test distribution of object placements to ensure tasks being feasible.
the agent needs to utilize a thin
Results. Table presents the planning time for all the evaluated methods. In scenarios with numerous objects available for interaction, searching directly for low-level manipulation primitives without the guidance of useful mechanisms can be extremely slow. However, using mechanisms as "macros" in
7

---

Method
Task
Task 2
Basis Ops Only
0%
0%
Ours (Macro)
20%
0%
Ours (Macro Sampler)
100%
90%
Task hold(Block) Task 2: suppkri(Block, Slope) Figure 6: Illustration of compositional tasks.
Table 3: Average success rate novel compositional tasks 10 runs. Timeout is minutes.
the search process significantly enhances efficiency, enabling the system to solve a greater number of tasks within the given time limit. The learned samplers further improve the overall efficiency.
Related Work
Approaches toward manipulation with primitives have been extensively studied in the field of robotics. These approaches aim to design algorithms that can perform complex tasks by decomposing manipulation tasks into a sequence of primitive actions, such as grasping, pushing, and placing. Task and motion planning algorithms provide methods for generating high-level task plans while considering the robot's physical constraints and the environment's 's dynamics. These approaches can be roughly grouped into two groups: sampling-buid -based such as HPN [9]. IDTMP [10], and many others [11, 12, 6], and global optimization- such as Looie-Geometric Programming [13] and Pang et al. [14]. Our method falls into the group of sampling-based
based planning, and extend existing approaches to handle briefly-dynamic tasks [3]. The major difference between our work and existing work in task and motion planning is that we focus on learning mechanisms that are sequences of basis manipulation operations and show significant improvement in complex manipulation tasks.
Our algorithm draws inspiration from contact-based modeling approaches in robot manipulation. In particular, various methods have been presented to make manipulation plans in the contact space, between rigid bodies as well as within robot hands [15, 16, 17, 18, 19, 20, 21, 22, 23J. Algorithms such as CMGMP [24, 25, 4] can automatically generate possible contact modes based on three primitive modes: fixed contact, separating, and sliding contact. In contrast to these methods that mostly focus on the characterization of contacts and planning with them, we consider learning mechanisms from contact modes single demonstration to solve new tasks efficiently.
Some mechanisms acquired by model have been traditionally deseribed tool-use skills [26, 27, 28, 29]. In comparison to these works, we presented novel framework for generating tool use trajectories with a planner based on contact sampling and showed that the learned mechanisms can be recombined with a general task and motion planner to solve more complex tasks. Recently, people have explored learning to predict tool-using trajectories or "object affordances" [30, 31, 32, 33, 34, 35), usually from a given set of demonstrations. By contrast, this paper focuses on generating trajectories with novel objects based only on a single demonstration, and combining the learned skills.
Limitations and Conclusions
Our mechanism learning framework faces two key challenges. First, obtaining detailed contact information from demonstrations can be difficult. This issue could potentially be addressed by integrating computer vision techniques capable of detecting contacts from videos. Second, the utilization of physical simulators for both learning and planning may not accurately replicate real- world physics, limiting the transferability of mechanisms real robots. Future work should explore real-world fine-tuning to tackle this limitation.
In conclusion, this paper has introduced a novel framework that enables machines to learn, reuse, and generalize manipulation strategies (i.e., the mechanisms) from a single demonstration and subsequent self-play in a distribution tasks. By characterizing each mechanism as a sequence of contact mode changes, the framework achieves notable generalization and compositionality. It generates successful trajectories for both novel object instances and categories under for the recombination of learned mechanisms solve complex tasks. Our framework can also be flexibly extended by incorporating other basis operations such compliance and forceful motion.
setting, and allows

---

Acknowledgement. We thank anonymous reviewers for their valuable comments. This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF- 123111,,,,,, NSF grant 2214177; AFOSR grant FA9550-22- 1-0249; from ONR MURI grant 400014-22- ARO grant 1N9-333-3-11-4 1-0034; the MIT-IBM Watson AI Lab; the MIT Quest for Intellienne;; and the Boston Dynamics Artificial Intelligence Institute. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
Bibliography
[1] R. Baillargeon and F. Delong. Explanation-Bane -asood Learning in Infancy. Psychonomic Bulletin Review, 24:1511-1526, 2017.
[2] A. Segre and G. Delong. Explanation Based Manipulator Learning: Acquisition of Planning Ability Through Observation. In ICRA, 1985.
[3] M. T. Mason. Mechunics of Robotic Manipulation. MIT press, 2001. 2,
[4] X. Cheng, E. Huang, Y. Hou, and M. T. Mason. Contact Mode Guided Motion Planning for
Quasidynamic Dexterous Manipulation in 3D. In ICRA, 2022. 2, 8
[5] C. Garrett, R. Chitnis, R. Holladay, Kim, Kaelbling, and T. Lozano-Pérez. Integrated Task and Motion Planning. Annual Review of Control, Robotics, & Autonomous Systems, 265-293, 2021.
[6] C. R. Garrett, T. Lozano-Péree and L. P. Kaelbling. Stream:mmm:: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning. In ICAPS, 2020. 4,
[7] Hoffmann and B. Nebel. The FF Planning System: Fast Plan Generation through Heuristic Search. JAIR, 14:253-302 2001.
[8] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets a Metric Space. NeurlPS, 2017.
[9] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In ICRA, 2011. 8
[10] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E. Kavraki. An Incremental Constraint- Based Framework for Task and Motion Planning. I/RR, (10(10)::134-1155) 2018. 8
[11] M. R. Dogar and S. S. Srinivasa. A Planning Framework for Kon-Prehensile Manipulation under Clutter and Uncertainty. Autonomous Robots, 33:217-236, 2012.
[12] Srivastava, Fang, Riano, R. Chitnis, S. Russell, and Abbeel. Combined Task and Motion Planning through Extensible Plannee-Independent Interface Layer. In ICRA, 2014.
[13] M. Toussaint. Task and Motion Planning. IJCAI, 2015.
Programming: An Optimization-Based Approach to Combined
[14] T. Pang, H. Suh, L. Yang, and R. Tedrake. Global Planning for Contact-Rich Manipulation via Local Smoothing of
Contact Models. arXiv:2206. 10787, 2022. 8
[15] J. C. Trinkle and J. J. Hunter. A Framework for Planning Dexterous Manipulation. In ICRA, 1991.
[16] X. Ji and J. Xiao. Planning Motions Compliant to Complex Contact States. IJRR, 20(6): 446-465, 2001. 8
[17] M. Yashima, Y. Shiina, and H. Yamaguchi. Randomized Manipulation Planning for a Multi- Fingered Hand by Switching Contact Modes. In ICRA, 2003. 8

---

[18] G. Lee, T. Lézano-Pérez, and L. P. Kaelbling. Hierarchical Planning for Multi-Contact Non- Prehensile Manipulation. In IROS, 2015. 8
[19] I. Mordatch, E. Todorov, and Z. Popović. Discovery of Complex Behaviors Through Contact- Invariant Optimization. ACM Transactions on Graphics (ToG), 31(4):1-8, 2012. 8
[20] K. Hauser and J.-C. Latombe. Multi-Modal Motion Planning in Non-Expansive Spaces. IJRR, 29(7):897-915, 2010.
[21] J.PP. Sleiman, J. Carius, R. Grandia, M. Wermelinger, and M. Hutter. Contact-Implicit Trajectory Optimization for Dynamic Object Manipulation. In IROS, 2019. 8
[22] B. Aceituno-Cabezan and A. Rodriguez. A Global Quasi-Dynamic Model for Contact-Trajecttog Optimization in Manipulation. In RSS, 2020. 8
[23] E. Huang, X. Cheng, Mao, A. Gupta, and M. Mason. Autogenerated Manipulation Primitives. IJRR, 2023. 8
[24] J. Xiao and X. Ji. Automatic Generation of High-Level Contact State Space. IRRR, 20(7): 584-606, 2001.
[25] X. Cheng, E. Huang, Y. Hou, and M. T. Mason. Contact Mode Guided Sampling- Baapling-Based Based Planning for Quasistatic Dexterous Manipulation in 2D. In ICRA, 2021. 8
[26] A. Stoytchev. Behavior-Groundee Representation of Tool Affordances. ICRA, 2005.
[27] B. Bril, R. Rein, T. Nonaka, F. Wenban-Smith, and G. Dietrich. The Role of Expertise in Tool Use: Skill Differences in Functional Action Adaptations to Task Constraints. Journal of Experimental Psychology: Human Perception and Performance, 36(4):825, 2010.
[28] C. Nabeshima, Y. Kuniyoshi, and M. Lungarella. Towards Model for Tool Tool-Body Assimilation and Adaptive Tool-Use. In ICDL, 2007. 8
[29] A. Gonçalves, G. Saponaro, L. Jamone, and A. Bernardino. Learning Visual Affordances of Objects and Tools Through Autonomous Robot Exploration. In ICARSC, 2014. 8
[30] A. Gupta and L. S. Davis. Objects Action: An Approach for Combining Action Understanding and Object Perception. In CVPR, 2007.
[31] Y. Zhu, Y. Zhao, and S.-C. Zhu. Understanding Tools: Task-Oriented Object Modeling,
Learning, and Recognition. In CVPR, 2015. 8
[32] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision. I/RR, 39(2-3): 202-216, 2020.
[33] Q. Li, K. Mo, Y. Yang. H. Zhao, and L. Guibas. IFR-Explore: Learning Inter-Object Functional Relationships in 3D Indoor Scenes. In ICLR, 2022.
[34] Z. Lai, S. Purushwalkam, and Gupta. The Functional Correspondence Problem. CVPR, 2021.
[35] K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From Pixels to Actions for Articulated 3D Objects. In CVPR, 2021. 8
10

---

Supplementary Material for Learning Reusable Manipulation Strategies
Table of Contents
A Method Detail
A.1 Planning Algorithm
A.2 Briefly Dynamic Manipulation
A.3 Basis Operations
12
A.4 Samplers for Generic Operations
12
B Experiment Detail
B.1 Mechanism Learning Setup
14
B.2 Sampler Learning
14
B.3 Physical Robot Deployment
16
Method Detail
A.1 Planning Algorithm
Algorithm shows the bi-level search algorithm we use. the discrete search level, we enumerate both basis operations well as mechanism operations. During the continuous parameter search phase, for basis operations instantiated from mechanisms, the mechanism-speciif sampler rather than the generic sampler for continuous parameters.
Algorithm 2 Bilevel Search Algorithm
procedure SF s0, 9, operator schemas)
2: plan_ges <- SymbolicSearch g. operator (schemrrssccemss) For all candidate sequences of basis operations
> Explore discrete plans
3:
for all plan plan_gen do CONTNNOUSSEEARCHIIO 9, plan))
4:
5: procedure CONTINOOSSSEARCHIE I plan))
6: grounded plan +- 0; 8 <- 80
7:
for all op E plan do for all arg & op.args do
8:
9:
arg InvokeSampler(op ..spppllerrr
Generate continuous parameters
10:
CheckPreconditionn(p) s) then
11:
Simulate the operator with sampled parameters. grounded plan <- grounded plan U {op)
op)
12:
13:
else break
14:
if IsGoal Achievee groudddd ggrondedd g) then
return grounded
Return the first plan that achieves the goal
return empty
A.2 Briefly Dynamic Manipulation
The system can handle robot-object and object-object contact without assuming quasi-static motion. For example, when placing objects surfaces, consider subsequent pose changes: objects placed on inclined surfaces may slide down, and heavy objects placed on levers can alter the pose of the

---

Before
After
Refore
fter
Case I: non-rigid attachments between objects while moving.
Case 2: object pose change after placement due to physics.
Case 3: support object pose change after new objects being placed.
Figure 7: Illustration of three brietly-dynamic manipulation scenarios in the paper.
(raction nove-with-cortacc sparameters (?teol iten Ptarget iten ?support: itan
Hand
?param contact-nove-parar 7qt trajectory) :precondition (and
Grasp:
(holding #tool)
Ladle
(support Itarget ?support)
?paran)
points, nornal
Contuct: (p. m)
(valid-come seliid-cntact-maae trajectory traajjecttyyy
?paran 2qt)) robot trajectory
:effect (and
(assign (rohot-configg) ...) 5; robot pasition
Spoon
(assign (pose Ptool). ...) 33 update tool position
Support: (p.
(assign (pose #target) ...) update target position
Figure 8: The modeling of the robot basis operation move-with-ccomtet using a STRIPS- syntax.
lever. Formally, we assume briefly-dynann manipulation setting, where the robot controller is position control-based, and manipulated objects may experience acceleration and velocity until they reach stable configuration. Figure 7 illustrates few examples of briefly-dynamic manipulation tasks handled by our sampler and planner.
A.3 Basis Operations
In our manipulation context, each schema represents either robot action that does not change the contact mode graph (e.g., moving the amn without object collisions) primitive action that modifies the contact mode (e.g., grasping an object). Table shows the complete list of basis operators used in this paper.
The precondition and the effect of an action schema describe the contact relationships between objects before and after the execution. Fig. showcases concrete example. The action schema involves three objects: the object being held ?tool, the target object that is in contact with the tool object Ptarget, and the object that currently supporting the target object ?support. Additionally, there are two continuous parameters: ?param specifies the contact between ?target and ?tool, including the contact surface and contact normal; ?qt specifies the robot arm trajectory as a sequence of joint-space waypoints. This action updates the robot joint angles, and the poses of ?tool and ?target. Given the discrete and continuous parameters, we use joint-space position controller execute the actions and use the execution results to update the state variables.
We will present the implementation details for the samplers associated with each operator in the next section (Appendix A.4). At a high level, these samplers are designed to be very generic: for grasping, it randomly samples two parallel surfaces on objects; for object-object contact, randomly samples surfaces on the object and then transforms the object held by the robot so that two surface normals point each other.
A.4 Samplers for Generic Operations
Recall that there three types of continuous variables be sampled for the basis operators described in Table 1: grasping poses relative to object (represented as SE(3) poses of the end-effector relative
12

---

to the object), placement poses (represented as SE(3) poses in the support object frame), contacts between two objects (represented as the SE(3) pose of object in the frame of object 2), and robot arm trajectories (represented as a sequence of arm trajectories). Here, we supplement the list of samplers we use to generate these continuous parameters. They are designed to be generic, relying solely on geometry and not specific object semantics (c.g., soup ladle grasping).
Grasp (G). The grasp sampler, G(O,T,) To), accepts the object's shape and current pose, O and To respectively, and identifies two "parallel" surfaces on the object mesh, represented as (P1, n1) and (p2, ne).. where P1 and p2 are two points and n1 and n2 are surface normals. The definition of being parallel is that: (pi- -22) #1 = I and R1 n2 = -1. It then computes a corresponding robot end-effector pose The such that Tge centered at the midpoint between P1 and Pa, and Tee is perpendicular to n1- It then checks the distance between two surfaces so that the parallel gripper can solver.
hold the object Te-- Finally, it checks the reachability of Tee using
Placement (P), The placement position sampler, P(O O2, To2), considers the shapes of both the holding object, O1 and the target support object, O2, and the pose O2. It randomly samples two surfaces, represented as (PI.21) and (p2: 122), one on each object such that n2 (0, 0, 1)² > 0.9 (i.e., n2 is close to the +2 direction). Next, it solves for transform on O₂ such that Tpi To2P2 and Tmi -T2212 (essentially place pl O p2 and pointing towards 112).
Object Contact (C', For both robot-object and object-object contact, the object contact sampler, C(O,, 22 Toz, O,,Ta takes five arguments, including the current holding object O (or the robot gripper itself when not holding anything). the object to contact O2 and its current pose To2, and the object that supports O2: O, and pose T.. It first randomly samples two surfaces, represented as (pI,n) and (p2, 22) on O and O₂ respectively. Since we do not consider pushing O2 "towards" the supporting object O,, additionally require that n2 is perpendicular to n,, which is the direction of the support force from O₂ O2. Next, it solves for transform T O such that Tpi To2P2 and Tni = -T2nnz (essentially place P1 on O at P2 and pointing towards n2 to excert force).
Trajectory (T). For grasping and placement trajectories, the trajectory sampler, T(Tinit; Traget, considers the initial and target end-effector pose of the robot gripper. It first uses an inverse kinematic solver solve for two robot configurations at the designated end-effector pose qiiir and Guarger Next, compute collision-fre free trajectory (except for collisions with the object being held and the object to contact) using Bidirectional Rapidly-exploring Random Tree (BiRRT) algorithm.
For witre--wth-contacc trajectories, the trajectory sampler, T(Tiiit P1, A1, P2, n22), accepts the initial configuration of the robot, finit, and the contact surfaces on the two objects sampled using the object contact sampler C: (p1,n) and (p2, n2). proceeds randomly sample "push" distance, along the contact normal direction, n1, from uniform distribution in the range [0.05, 0.25] meters. Subsequently, the sampler generates the arm trajectory by invoking the BiRRT algorithm to follow set of waypoints corresponding to linear Cartesian-space motion along n. by distance d.
13

---

B Experiment Detail
B.1 Mechanism Learning Setup
Our evaluation encompasses six distinct mechanisms, grouped into two categories: the first four tasks assess "tool-use."
(Edge) pushing objects to the edge of table for pickup. There are four object models used in this mechanism: plate, calculator, caliper, and document.
(Hook) using tools to reach for distant objects. There are five objects that can used as the "hook:" wooden L--hape stick, soup ladle, hammer, spoon, and caliper.
(Lever) flipping objects using heavy objects levers. There are four "heavy" objects that can be used to flip the plate: box, spoon, dipper, and walnut.
(Poking) using tools to poke objects out of tunnel. There are three object models that can be used as "poking" tool: wooden stick, spatula, and spoon.
The remaining two tasks fall under the "reasoning about stability" category.
(Center-of-Mase) achieving stable object placement on another object. There are three object models to be placed on the small block: plate, calculator, and document.
Baane-and-Blockeer using objects blockers prevent objects from falling off inclined surfaces. There are three object models that can be used the blocker: wooden stick, wooden L-shape stick, and spoon.
For each environment, we first manually defined canonical pose for each object such that the mechanism is feasible. Next, for each training and testing instance, we randomly apply small translations (a uniform distribution within + 5 centimeters) and small rotations (uniform within +: 15 degrees) the canonical pose of each movable object.
B.2 Sampler Learning
Taking a closer look at the importance of sampler learning, Fig. 9 illustrates breakdown of the number of samples required for the "hook use" mechanism using our planning algorithm, with the generic sampler and with the learned sampler. Fig. 3b shows the inferred macro definition for this mechanism, and here count the number of samples produced by each individual sampler. In this case, most of the samplers produced to generate candidate grasping poses of the tool and possible contacts between the tool and the target (i.c., how to reach the tool).
14

---

Grasp of Tool
Generic Smapler
Learned Sampler
Grasp Trajectory of Tool
Tool-Target Contact
Tool Moving Trajectory
Placement Tool
Placement Trajectory
Grasp of Target
Grasp Trajectory of Target
20
40
100
120
Generated Samples
Figure 9; Breakdown of samples produced by different samplers for the hook using task.
15

---

B.3 Physical Robot Deployment
Franka Emika Panda
Frame- mounted Camera
Used objects in the experiments
Figure 10: Our world setup.
Task
push-to-edge(plate)
hook-use(banana, (ball. apple))
Suce, Rate
80%
70%
Table 4: Success rate of our physical robot setup on two representative mechanisms.
Our real-world setup, shown in Fig. 10, contains Franka Emika Panda robot arm with a parallel gripper, mounted table. We also have Intel Realsense D435 camera mounted on the table frame, pointing 45 degrees down. Our vision pipeline contains six steps: first, based on the calibrated camera intrinsics and extrinsics, we reconstruct a partial point cloud for the scene. Second, we crop the scene to exclude volumes outside the table (e.g., background drops, etc.) Third, we use a RANSAC-based algorithm to estimate the table plane, and extract all object point clouds on top of the table. Fourth, we use Mask-RCNN to detect all objects in the RGB space, and extract the corresponding point clouds. For objects that not detected by the MaskRCNN, to cluster their point clouds, and then run the SegmentAnything model to extract their segmentations in 2D, and subsequently the point clouds. Finally, we perform object completion by projecting and extruding the bottom surface for all detected objects down to the table.
first use DBScan
Next, we import the reconstructed object models into our physical simulator PyBullet, and directly deploy our planning algorithm with the learned mechanisms to compute plans. Since we the same robot model in simulation, practice, we execute in a closed-loop manner. If the grasp fails (which can be detected by the gripper sensor), we move the arm to its neutral position and replan. After each placement action and pushing action, we use the vision pipeline to obtain an updated world state and replan.
directly execute the planned robot trajectories in the real world. In
We demonstrate and evaluate the system on two example tasks: hook-using for distant objects. We use the same of objects on the table but with different initial configurations. We repeat each experiment 10 times and report the success rate. Table summarizes the result. The visualization of our vision pipeline and the robot videos can be found on our website.
pushing plates to the edge and
The push-to-edge task is relatively easier given the learned mechanism. The only two failure cases in our experiment were triggered by the robot pushing the plate too far and the plate falling off the table. The hook-using task is more challenging. There are two main failure modes we observe in the experiments. First. the planner favors grasping objects (c.g.. the banana "hook") at the tip, which is very unstable grasp sometimes causing the executor to fail. Also, sometimes the planner generates
16

---

wrong object motion for the objects because 3D of reconstruction errors: the planner is using "hallucinated" part of the object to push the distant object-currentl, we cannot recover from this type of failure. Finally, sometimes the sampled grasp is invalid, again because of the errors object perception. In this case, our close-loop execution improves the performance.
17


Reviews:
[{"review_id": "hotodEzrqxG", "reviewer": "Reviewer_KXeh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces a novel method for learning reusable manipulation strategies. The core functionality of the proposed approach is the ability to sequence single demonstrations into smaller, core skills that can be sequenced together. The individual skills are then learnt through trial and error and by using self-play. Overall, the paper is well written and clearly discusses both pros and cons of the proposed approach. The evaluation, conducted in simulated environments, is extensive and well done. My only big concern, also acknowledged in the limitations section, is that this method is not straightforwardly generalised to physical systems. Having said that, I find this work as an excellent paper but question its fit for CoRL.", "summary_of_recommendation": "Overall, I like this approach but I am not convinced it is a great fit for the CoRL community. Having said that, it seems like a well executed paper and as a result I recommend a weak reject with mild confidence purely on the basis of its fit to the CoRL community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why was this approach not tried on a physical system, substituting some of the skill learning techniques with more classical approaches and/or approaches that rely less on online learning can alleviate a lot of the constraints\n- How can this approach scale to more complex dexterous tasks, such as insertion with tight tolerances, drawer opening etc", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths and Weaknesses\n### Strengths\n\n - An automated method for learning reusable strategies from limited number of demonstrations\n - An answer to an important question of how to build reusable manipulation skills\n - A well written paper\n\n### Weaknesses\n - Unclear scalability to physical systems\n - Convoluted algorithm that does not necessarily scale to dexterous tasks", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "hlsAuN1CBvK", "reviewer": "Reviewer_W6qd", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a framework for learning manipulation skills, which they refer to as mechanisms, and are composed of sequences of defined basis manipulation operations. The problem of manipulation is looked at as a STRIPS planning problem. As a result, even a single user demonstration of a task can be decomposed into parametrised manipulation operations and then repeated. The author present results from their method in simulation on a series of manipulation tasks of varying complexity.", "summary_of_recommendation": "I recommend weak reject but would be willing to change my recommendation during the rebuttal period.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* how much effort do you think it will be to have this method run on a real robot? Would it still be able to support 1-shot learning or maybe it would need more demonstrations per task?\n* what happens if there are distractor objects in the scene - is the method capable of figuring out what matters in a scene and what doesn't with its current STRIPS formulation?\n* fixing the set of basis operators ultimately means that the method won't be able to support certain skills - what behaviours can't the robot learn and do?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a problem that is relevant for the robot learning community - being sample efficient and being able to generalise well in long-horizon tasks. \n\nStrengths:\n* the authors choose to look at learning from demonstration as a hybrid system which ultimately means representing some of the information encoded in the user demonstrations as symbols. As a result they can utilise off-the-shelf proven planning algorithms and planning languages which in turn allows them to be very efficient and to generalise to novel skills, even non-demonstrated ones.\n\nWeaknesses:\n* the method is heavily dependant on having state information from the simulator about contacts, in order to be able to decompose a single demonstration into its subparts.\n* the paper is generally hard to read and introduces a lot of it's own naming and terms that are not clearly explained and examplified - i.e. manipulation skills are mechanisms, briefly-dynamic, learned specialised samplers (I'm still not sure what are the samplers even after reading the paper a few times.)\n* all experiments are solely done in simulation, partly because of the method is too dependent on having full state observability. It is unclear whether and how much the method would succeed in scenarios where that's not the case - i.e. the real world with physical robots, noisy sensors and very-hard-to-estimate contacts purely from vision.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ZfFIwqMSQl", "reviewer": "Reviewer_5d1G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present an approach for learning manipulation mechanisms. A mechanism is a reusable sequence of \"manipulation primitives\" that comes with an individual sampling strategy for each primitive. While the primitives seem to be predefined, the learned manipulation mechanisms can be reused in other learning setups. Individual primitives and mechanisms are parametrized such that they are not limited to single tools but can also be generalized to setups which are different from the single demonstration that was used to learn them.", "summary_of_recommendation": "I chose my rating mainly because I did not find any major flaws in the paper. I don't think my major concerns and questions about the approach in general have to be tackled in this paper, but can also be follow-up work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why is a comparison with other state-of-the-art approaches not feasible? Why did you choose your own baseline over existing approaches?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- One major weakness which is also discussed in the limitations section by the authors are the missing real robot experiments.\n\n- I understand that the presented concept is sampling based and can be integrated into sampling-based task planners. I would like to see a clarifying statement on why a comparison to existing approaches is difficult, since in the exerperimental section the presented method has only been compared to a baseline approach.\n\n- As also discussed by the authors uncertainty and imprecise demonstrations are a major issue with the approach. However, I also believe those topics can be tackled by follow-up work. Additionally, the aforementioned difficulties are shared with other planning approaches and are not specific for the presented approach.\n\n- Line 154: they present -> they are present", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "TrKmX-pU0J", "reviewer": "Reviewer_sZoG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The submitted paper presents a new framework for learning reusable manipulation \"mechanisms\". These mechanisms can be learnt from a single demonstration and then additional self-play. A mechanism learns the sequence of contact mode changes between the robot, objects and support surfaces. Planning with mechanisms assists with efficiently creating long-horizon plans.", "summary_of_recommendation": "The authors have presented a paper that introduces a framework for learning \"mechanisms\" that are composed of lower-level basis operations. The idea leverages the sequence of contact mode changes in a demonstration to efficiently learn. Overall the work is well presented although is missing some discussion on the limitations of the proposed method and the requirements for the demonstration. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "What is the limit to the number of basis operations that can be composed in a single mechanism? Are there downsides of composing long strategies? \n\n\nOnly 100 samples are used to train each scoring classifier for predicting successful samples in simulation with limited objects and starting configurations. This seems remarkably few. Scaling the scoring classifiers to real-world environments appears like it will become prohibitively expensive. Do you believe the method, on top of the other obvious sim-2-real issues, will be able to scale in this way to transition to real-world setups? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The paper presents a new framework for learning a useful manipulation strategy composed of multiple basis operations (or lower-level skills) that utilises the change in a contact mode graph to understand the transition between basis operations. \n- The idea I believe is unique and is framed in a way such that it is useful for long-horizon task planning where there are also continuous variables (task and motion planning).\n- The authors propose a method for learning new mechanisms from a single demonstration and then utilising self-play to improve the generalisation of the mechanism to a known set of objects.\n- The paper is generally well written with ideas clearly and logically presented although there are points where refinement could help to avoid confusion. For example, the delineation between skills, tricks, mechanisms, strategies, basis operations etc. could be made clearer and earlier. \n\nWeaknesses:\n\n- The authors fail to compare their method against any competitive or state-of-the-art baseline. An option here might be a hierarchical RL policy seeing as there are a fixed set of objects and all experiments are completed in simulation. \n- There is mention that this method is able to handle briefly-dynamic tasks, however, there is no proof of this in the main body of the paper and seeing as planning and execution is completed in simulation it is difficult to see how this would remain true if transferred to real-world environments. \n- The limitations of devising a manipulation strategy as a mechanism are not explored nor are the requirements for the demonstration. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents an approach to learn reusable skills, referred to as "mechanisms". These mechanisms learn sequences of contact mode changes between the robot and objects based on a single virtual demonstration and subsequent self-play. While the evaluations in the manuscript were only performed in simulation, the authors provide real world experiments during the rebuttal period. Unfortunately, the approach still requires an initial virtual demonstration even for the real world experiments, which is not ideal but I it should not be a disqualifying limitation.

The authors generally did a good job in the rebuttal and I believe given the revisions described in the rebuttal that this work can be accepted for CoRL 23.
Decision: Accept (Poster)