Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Authors: Xingyu Lin
Carl Qi
Yunchu Zhang
Zhiao Huang
Katerina Fragkiadaki
Yunzhu Li
Chuang Gan
David Held
Venue: CoRL 2022 Poster (2022)
Keywords: Long horizon planning
deformable object manipulation

Abstract: Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on short-horizon tasks or make strong assumptions that full-state information is available, which prevents their use on deformable objects. In this paper, we propose PlAnning with Spatial-Temporal Abstraction (PASTA), which incorporates both spatial abstraction (reasoning about objects and their relations to each other) and temporal abstraction (reasoning over skills instead of low-level actions). Our framework maps high-dimension 3D observations such as point clouds into a set of latent vectors and plans over skill sequences on top of the latent set representation. We show that our method can effectively perform  challenging sequential deformable object manipulation tasks in the real world, which require combining multiple tool-use skills such as cutting with a knife, pushing with a pusher, and spreading dough with a roller. Additional materials can be found at our project website: https://sites.google.com/view/pasta-plan.

Planning with Spatial and Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin *f
Carl Qi #f
Yunchu Zhang
Zhiao Huang
Katerina Fragkiadaki f
Yunzhu Li
Chuang Gan
1
David Held
Effective planning of long-horizon deformable object manipulation requires suitable abstractions both the spatial and temporal levels. Previous meth- ods typically either focus short-horizon tasks make the strong assumption that full-state information is available. However, full states of deformable objects are often unavailable. In this paper, we propose PlAnning with Spatial and Temporal Abstraction (PASTA), which incorporates both spatial abstraction (reasoning about objects and their relations to each other) and temporal abstraction (reasoning over skills instead of low-level actions). Our framework maps high dithesinnnion 3D point clouds into of latent vectors and plans skill sequences with the latent set representation. Our method can solve challenging, novel sequential deformable object manipulation tasks in the real world, which require combining multiple tool-use skills such as cutting with knife, pushing with pusher, and spreading dough with a roller. Additional materials can be found on our project website."
Abstract:
Keywords: Long-horizon Planning, Deformable Object Manipulation
cutter
pusher
roller
Execution
Planning
Cut
Push
observed latent vector
More
Steps
latent subgoals
latent subgoals
Figure 1: Long-horizon dough manipulation with diverse tools. Our framework is able to solve long-horizon, mool. deformable object manipulation tasks that the agent has not seen during training. The illustrated task here is to cut piece of dough into two with cutter, transport the pieces to the spreading area the left (with high-friction surface) using pusher, and then flatten both pieces with roller.
Introduction
Consider a typical cooking task of making dumplings from dough. People plan over which piece of dough to manipulate and which tool to use in sequence, incorporating both spatial and temporal abstractions. spatial abstraction reasons about objects, parts, and their relations to each other, such as reasoning about pieces of dough instead of reasoning about individual dough particles; such
equal contribution: i CMU UC San Diego: & Stanford University; UMass Amherst & MIT-IBM Lab htttp://ssites google cwm/ http://vee/paataapppee -lan
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

spatial abstraction enables efficient planning and compositional generalization. On the other hand, temporal abstraction incorporates abstract actions represented as a set of skills. With abstract actions, the agent plans for the types and parameters of the skills to execute over a period of time, instead of making plans for low-level actions such as joint torques at each time step. Temporal abstractions allow planning the skill level, enabling more efficient optimization for solving long-horizon tasks. An autonomous robot that operates in unstructured environments should be able to reason about world dynamics using high-level spatial and temporal abstractions instead of reasoning only over the physical state, raw sensory observation, or low-level robot actions.
The research community is making rapid progress towards developing state abstractions for manipu- lating deformable objects, including key points [1. 2]. graphs [3, 4], dense object descriptors [5], or implicit functions [6. 7]. However, most of these approaches do not make abstractions the temporal level, limiting their use short-horizon tasks. Methods are also being developed with temporal abstractions, planning over set of skills to solve long-horizon tasks [8, 9, 10]. However, the lack of spatial ubstraction severely limits their generalization ability. Therefore, it remains key question in robot learning on how to learn spatial and temporal abstractions within a unified framework for complex and long-horizon manipulation tasks.
In this work, focus on the challenging task of sequential deformable object manipulation, as shown in Figure 1. We consider set of dough manipulation tasks that require sequentially applying different skills using multiple tools to manipulate dough, such as spreading using a roller, cutting using a knife and pushing using pusher, where the longest task requires applying 6 skills in sequence. Deformable objects like dough have nearly infinite degrees of freedom. As such, in this work, we dynamically cluster points in a point cloud into different groups and learn a point cloud encoder to map each element in the group into latent vector. In this way, we obtain a compositional 3D set representation of the state space. Given observation and target point cloud, we then sample skill sequences with subgoals generated in this latent space. We learn skill abstraction modules to determine the feasibility and score of each skill sequence and use them for planning.
Our contribution of this paper is a framework that PlAns with Spatial and Temporal Abstraction (PASTA) by learning a set of skill abstraction modules over a 3D set representation. Our framework can compose a set of skills to solve complex tasks with more entities and longer- horizon than what was seen during training. We show that PASTA significantly outperforms ablation that performs planning with a flat representation without spatial abstraction (e.g. without set representation). Finally, our planner can be trained in simulation and transferred to the real world.
2 Related Work
Model-based Planning for Sequential Manipulation. One line of research for sequential manipu- lation is Task and Motion Planning (TAMP). TAMP systems typically assume known object states and known effects for the action operators [11, 12, 13, 14, 15]. However, it is difficult to estimate states and dynamics for unknown objects from partial observations. While recent works have made progress in learning certain components of the system, such as the logical states [16] from high dimensional observations or learning action models [17, 18, 19] from interactions, they still require either known states or known action operators. In contrast, we do not assume known states or action operators, and learn 3D set representation as well as the action model with the representation.
Another approach learns dynamics directly from visual observations [3, 20, 21]. Most of these works focus on learning a one-step dynamics model for planning short-horizon tasks. A few works learn the dynamics model over set of skills and use it for sequential manipulation of rigid objects [22, 23] or deformable objects [8]. However, these works do not use an object-centric representation and thus cannot easily generalize to more complex scenes. In contrast, our framework unifies both temporal and spatial abstraction and can perform long-horizon manipulation for complex tasks with more objects than in previous work, as we will show.
Planning with Spatial Abstraction. Prior works leverage spatial abstraction to facilitate solving tasks that involve complex dynamics and high-dimensional observations. These works either model
2

---

Generate demonstration dataset
(b) I Learn set skill abstraction from demonstration
Demonstration Trieectories each skill
feasibiliity predictor
observed pe tool
cut
spread
push
PointNett+
set cosl predictor
set policy
(c) Planning and execution with set skill abstraction
goal pc
M1,0
push Rassiiility
Mgg0
cost predictor
My
Jeusibility
push fassibility
pont
push
push
pobs
Step
Step 2
Step
ponni
Figure Overview our proposed framework PASTA. (a) first generate demonstration trajectories for each skill in a differentiable simulator using different tools. (b) then sample point clouds (pc) from the demonstration trajectories to train our set skill abstraction modules. (c) We map point clouds into latent set representation and plan over tool-use skills perform long-horizon deformable object manipulation tasks. pobs, pooal are the observation and larget pe; ui, denotes component, at step The example shows our method performs the CutRearrange task, which requires cutting the dough into two pieces with a knife and transporting each piece to its target location.
compositional system with Graph Neural Networks (GNN) [3. 4, 7, 24, 25] or learn policies directly from object-centric representations [26, 27]. These works demonstrate compositional generalization, but they learn policies or one-step dynamics models for planning, which can be difficult for solving long-horizon tasks. In contrast, our framework connects temporally extended spatial abstractions with a feasibility predictor to plan over longer time horizon. Xu et [28] learns a planner grounded on object-centric visual observation and can solve long-horizon tasks. However, planner takes symbolic goal and plans in predefined symbolic domain output symbolic subgoals. In contrast, method does not require defining symbolic planning domain.
Deformable Object Manipulation. Deformable objects have nearly infinite degrees of freedom and complex dynamics, making them very challenging to manipulate. Previous works have explored pouring liquid [6, 29, 30], rope manipulation [31, 32), and cloth manipulation [33, 34, 4, 35, 36]. Other papers have also explored manipulating elasto-plastic objects such deforming them by grasping [3, 37], rolling [38, 39, 40], cutting [41]. However, these works mostly only consider manipulation with skill at time. In contrast, consider the task of sequential manipulation using multiple tools. The one exception is DiffSkill
[8], where multiple skills are chained together. However, DiffSkill uses RGB-D images to represent the scene. In contrast, we use 3D set representation that separately encodes each entity in the scene, enabling compositional generalization tasks with more objects and longer- horizon. Furthermore, we use point clouds as the input and we are able transfer our planner from simulation the real world.
3 Method
Given a point cloud of the dough pabs and goal point cloud pgoal our objective is to execute sequence actions ..,..., aT... that minimizes the distance between the final observed point cloud and the goal D(Pphs pgoal) where pohs is the observation point cloud at time Tiot- We aim to solve long-horizon tasks that require chaining multiple skills in novel scenes with more objects than training. To do so, we present general framework that incorporates spatial and temporal abstractions, summarized in Figure 2. We point clouds input to all modules to enable easier transfer from simulation to the real world and enable robustness changes in viewpoint.

---

We assume access to an offline dataset of demonstration trajectories Ddemo from K skills, where each trajectory demonstrates one of the skills using one tool. We can learn skill policies by imitation learning from these demonstrations. To chain these skills to solve long-horizon tasks, we train a set of skill abstraction modules (Sec. 3.3) which can be used for planning the latent space (Sec. 3.1).
3.1 Spatial Abstraction from Point Clouds
First, we describe our spatial abstraction of the point cloud observation. Given a point cloud P E RNx33 first cluster the points into different components based on their proximity in space. In this paper, we apply DBSCAN [42] to P and group points into a set of entity point clouds {P; E RNNx33 Ji=t...C by separating points from high-density regions into different clusters. While other works on scene decomposition can also be used [43], we find this simple method to be sufficient for our tasks.
Scene Decomposition:
Planning directly in the digh-dimensional space of point clouds is inefficient. To enable efficient planning in latent space, we train point cloud variational autoencoder (VAE) [44]. The VAE model includes three modules: A point cloud encoder Ø IR N33 + Ll that maps each entity point cloud to latent vector, decoder &े : 24 - RN x3x3 that maps from latent space back into a point cloud, and prior distribution over the latent space Pa : 4 + [0. 1] which can be used to generate samples from the latent space during planning. We can then encode the point clouds {P:} into a set of latent vectors: {ur our set latent representation. We further achieve translation invariance by separating the translation from its shape embedding. See the Appendix for details.
Entity Encoding:
3.2 Learning Skills by Imitation
Given the demonstration trajectories Ddemo of K skills, we first distill these trajectories into K closed-loop policies. The input to the policy for the 1tth skill Ty is a subset of the observed point clouds { and goal point clouds { and a tool point cloud Phoot The policy only sees a subset of the dough point cloud and goal point cloud the scene to enable compositional generalization to scenes with more objects. For example, a mough--preading policy will only see the dough being spread. To achieve this, we train each set point cloud policy with behavior eloning and hindsight relabeling [45] on the demonstration dataset with attention mask that filters the non-relevant entity point clouds. During planning, this attention mask will be provided by the planner. The policy outputs an action at each timestep to control the tool directly.
3.3 Neural Spatial and Temporal Abstraction
We assume that each skill learned from the demonstration is only capable of performing a single-stage task with single tool for a single object. To solve longer-horizon tasks, we further learn feasibility predictor and cost predictor. They can be used to plan subgoals that chain the skills into a sequence, such that each subgoal is feasible for the corresponding policy to reach and the final subgoal reaches given goal. Additionally, all these modules take in the set representation (P.... Pc} as input to achieve compositional generalization.
Set Feasibility Predictor Similar to DifiSkill [8], we train feasibility predictor (u US) for each skill, where U {uf) I= U9 b= are latent set representations of an observation and goal point cloud respectively. The feasibility predictor outputs a value in [0, 1] denoting if the goal can be reached from the observation by executing the 4thh skill. In DiffSkill, the feasibility predictor uses a flat representation that takes in a single latent vector for all objects in the scene as input. However, as our skills such as cutting or spreading only need to take in a subset of the objects as input, we use the same attention method for the feasibility and assume that the feasibility predictor only takes as input a subset of the full set representation i C U ② C US, where Ü = (uf)i=i...., ih (u,) j= Here, N₂ and M₁ are the number of components in the observation and goal for skill k. The number of components in the observation and goal can be different since the number of components change before and after executing a skill; for example, the cut skill takes one component observation and cuts it into two components. As another example

---

for robot assembly [46], the number of entities increases when piece is disassembled into parts and the number of entities decreases when the parts are assembled. In this work, we manually define the number of entities N₁. Mr per skill. Determining which subset to attend to when executing each skill can be difficult; we make this decision during planning and defer the details to Sec. 3.4. We parameterize f& to be invariant to permutation using max-pooling layers. See Appendix for details.
We train the feasibility predictor of skill k with positive examples Ú Ú9, where the goal Ú79 can be reached from the observation Ü within T' timesteps by executing skill k. The negative examples are goal U that cannot be reached from the observation Ü using skill k. During training, we obtain positive pairs for the feasibility predictor by sampling two point clouds (Pobe, pgoa)) from the same trajectory in the demonstration set. To find Ü ÜS, we first cluster the observation and goal point clouds into two sets { { , (P.)) respectively. Then, we match point clouds in the observation set to those in the goal set by finding the pairs of point clouds that are within Chamfer distance of {(PP, P.)) Dchamfer (Py, P.) < <e). We then remove these point clouds from the corresponding set, since these are the point clouds that have already been moved to the target location in the goal. We can then encode the remaining point clouds into th, (79 as explained above. We generate hard negative samples by replacing one entity in the positive examples with a random latent vector.
Set Cost Predictor As we do planning in a latent space, we train set cost predictor as our planning objective which determines how close plan is to given goal. The set cost predictor C' takes two latent set representation as input U U2, Since our tasks focus on matching each entity in the observation with one in the goal, we assume they have the same number of components, i.c. IU = | = Ne. To compute the cost, we try to find the matching entity with the minimal matching cost: C ((u)), (u(){(()) = arg min, ΣN=, co co(u:, "(())) where a is permutation and og is cost prediction network parameterized by MLP trained to predict the Chamfer Distance between the point clouds corresponding to the two latent vectors. This allows faster planning compared to first decoding latent vectors to point clouds and then computing their distance. Finally, optimization of the cost is done by performing Hungarian matching between the two sets containing latent vectors.
3.4 Planning with Set Representation
Given an observation and a goal point cloud pobs pgoul, we plan for the types of skills to apply in sequence, the attention for each skill (i.e. find &o c U?), and the latent subgoals for each skill (i.e. the exact value for each latent vector in Ú'). As the simplest approach, we run a three-level nested optimization: In the top level, we exhaustively search over the combinations of skills to apply at each step, i.e. k1..., kin, where kn indexes the skill applied at the high-level step h. We only keep the sequences that end with the same set cardinality the goal by ensuring that E Mr, NNa N₂ No. where Ma, and Na,, are the number of observation and goal components for the skill applied at step h and No and Ng are the number of components in the observed and target point clouds.
In the second-level optimization, we search over different attention structures. Denote the latent set at the high level step h to be Uh. We formally define the attention structure step h be 7h which consists of a list of indices, each of length NNa, such that 1 selects a subset from Uh-1 to be the input to the feasibility predictor, i.e. Üth U/ C Uh-1, Assume that we have No. components before applying skill ky, i.e. | = N₂, and skill k2 takes K,, components as its observation. We can search over all C combinations of attention structures. For components not considered by the N₂ skill, its latent vector will remain the same at step h. The combination of each skill attention yields an attention structure I for the whole plan, as illustrated in Fig. 2(c). For this level of optimization, we use a sampling- based procedure to avoid an exhaustive search over topologically equivalent attention structures. See the Appendix for how we do this efficiently.
In the low-level optimization, for each attention structure I, we follow the optimization in DiffSkill [8]. We first sample multiple initializations for the set of latent subgoals U, where each latent vector in the set is initialized from our generative model. We can then perform gradient descent to further optimize the latent subgoals on the following objective:
5

---

H arg mig (k, U) = II fun (0h-1,0*)) k.I.U
,U)),
(I)
where k is the skill sequence, I is the attention structure of the plan, U the set of all latent subgoals, T = {u, Mi,,...Ma, are the latent subgoals at step h, 8 t f U is the attended observed set, and U is the goal set. Finally, we can use our policy to execute our plan by following each subgoal.
summary of our method can be found in Algorithm
Algorithm 1: Planning with Spatial and Temporal Abstraction (PASTA)
Input :emonstration Dataset Deemo skill horizon T, planning horizon H. modules for neural skill abstraction Tk: Jk, ra, Point Cloud VAE with encoder s, decoder v, prior pa for each valid skill sequence .,.... kn do for each valid attention structure r',.. .. do Initialize different latent subgoals ,'.... Ū from Pa. Optimize latent subgoals ('.....' . according to Sec. 3.4 obtain cost J(k, I, U)
2
3
Choose skill sequence k, attention structure and subgoals U that minimizes J(k, I, U); 6 for h + 0 to 1 do
Decode the subgoal from &h using the decoder yb:
7
Execute policy NA, following the subgoal;
Experiments
Our experiments are categorized into three parts: In Sec. 4.1, we describe the experimental setups and the baselines we consider. In Sec. 4.2 and Sec. 4.3, we show that PASTA outperforms the baselines and ablate different components in our framework. Sec. 4.4, we demonstrate that PASTA can be effectively transferred to the real world without any fine-tuning.
4.1 Simulation Tasks and Baselines
Environment setups We consider several long-horizon dough manipulation tasks and divide them into two categories. First, we consider the three tasks from Diffskill [8]: LiftSpread, GatherTransport and CutRearrange. These tasks require the agent sequentially compose at most two skills to spread, transport the dough. We further propose two new generalization tasks: CutRearrangeSrree (CRS) and Tes-Twice, where there are more entities during testing than during training. Similar to prior work [8], we specify the minimal planning horizon required for each task. Our approach also succeeds when we increase the horizon up to twice as long. See the Appendix, Sec 4.5 for details.
Generalization tasks The CRS task provides number of demonstration trajectories performing one of the three skills: Cutting with knife, pushing with pusher, and spreading with roller. The demonstration of cach skill only shows tool manipulating single piece of dough. During testing, the agent needs to cut dough into two, transport one piece to spreading area and then spread it. Generalization to more entities is required as there will be two entities in the scene, whereas during training there was only one entity. Can we do even more generalization? In the CRS-Twice task, we use the same agent trained on the CRS dataset and ask to cut two pieces of dough from chunk, transport both of them to a spreading area, and spread them both. This 6-horizon task with up to 3 entities in the scene, much more complex than the skill demonstrations the agent is trained on. nature of CRS- CRS-Twice, we specify the skill skeleton and use receding horizon
Due to the long-horiag planning for all the planning-base based methods. See the Appendix for details.
Baselines We consider several baselines in simulation: First, a gradient- based trajectory optimizer with oracle information (Traj-Opt) which can solve single-stage tasks for deformable object manipu- lation shown in prior works [47]. Second, model-free RL with Soft Actor Critic [48] with RGB-D image input (SAC-Image) Third, SAC agent that takes in the dough, target dough, and tool point clouds as input, the same our method (SAC-Point). Fourth is DiffSkill, a model-based planning

---

method from Lin er al. [8], which takes RGB-D images as input and has no spatial abstraction. The last one is Flat 3D, which extends DiffSkill to use 3D point clouds as input, with a "flat" 31) representation that encodes the whole scene to a single latent vector without any spatial abstraction.
Metric We specify goals as 3D point clouds of different geometric shapes such as boxes and spheres at specific locations. We report the normalized decrease in the Earth Mover Distance (EMD) approximated by the Sinkhorn diverge [49] computed as s(t fom where so, 8, are the initial and current EMD. We additionally set a threshold for the score to determine the success of trial.
#o
4.2
Comparison with Baselines in Simulation
Task (Horizon)
DifiSkill tasks GatherMove (2)
Generalization tasks
Method
LiftSpread (2)
CutRearrange (3)
CRS (3)
CRS-Twice (6)
Traj Opt (Oracle) [47]
0.818/ 40%
0.403 /
0.511 20%
0.312
0.227 / 0%
SAC-Image [48]
0.797 0%
0.567 / 20%
0.103 / 0%
0.562 / OK%
0.365 / 0%
SAC-Point [48]
0.796 0%
0.603 / 40%
0.147 /
0.573 / 0%
0.353 / 0%
DiffSkill Image [8]
0.920/ / 100%
0.683/ / 60%
0,249/ 20%
40.505 OS.
Flat 3D (Ours)
0.797 60%
-0.712 0%
-0,108 0%
PASTA (Ours)
0.904/ 100%
0.715 100%
0.837/ 80%
0.896
0.604 40%
Table 1: Normalized improvement and success rate of all methods two sets of tasks: tasks in DiffSkill and tasks that require generalization more steps and entities. For CRS and CRS-Twice, training data only contains skills operating on one component of dough but at test time there are more than two components. Only the hest-performing baselines CRS are evaluated on CRS-Twice. For LiftSpread and GatherMove, we consider the whole scene as a single spatial abstraction, so Flat 3D is equivalent to PASTA.
Table shows the quantitative results of simulation tasks. First, we show that using a 3D representation beneficial to planning and complex manipulation, as PASTA matches DiffSkill in LiftSpread and outperforms it in all the other tasks. Second, we highlight PASTA's compositional generalization power in CRS and CRS-Twice, in which there are additional components of dough at test time. Effectively using spatial abstraction to model the scene, PASTA achieves 100% success rate in CRS and retains a good performance in CRS-Twice. All the baselines, especially the planning baselines (DiffSkill, Flat 3D) fail dramatically, as they can only produce plans that consist of scenes seen in training. Impressively, PASTA is the only approach that reaches a non-zero success rate on these tasks that require compositional generalization. PASTA has a computational complexity of O(K |I| TG), where K is the number of skills, H is the planning horizon, [I] is the number of attention structures and To is the time it takes to solve each gradient-based optimization. We discuss more efficient planning algorithms in the limitation section well as in the Appendix, Sec. 5.
4.3 Ablation analysis
Ablation Method
Performance/ / Success
No Hard Negatives Feasibility
0.740/ / 40%
Table 2 shows the quantitative performance of each ablation and PASTA in CutRearrange. First, we consider a variant of feasibility predic- tor's training, which removes the hard negative samples and only uses random negative sampling (No Hard Negatives), which halves the success rate. Second, we consider two variants of the planner, one without gradient-descent (No Gradient Planning) and one without sampling (No Sampling Planning). The results show that both components are crucial to planning. We provide more ablations on the policy and the planner in the Appendix.
No Sampling Planning No Gradient Planning
-0.455 / 0%
0.329 0%
PASTA (Ours)
0.837 80%
Table 2: Ablation results from CutRearrange.
4.4 Real World Experiments
Figure shows our real world setup. We use a Franka robot with an Azure Kinect camera capturing the RGB-D observation of the workspace. The robot is equipped with a tool station [50] that allows an automatic change of tools. For real world "dough", we use Kinect Sand as a proxy because of its stable physical property. We transfer the feasibility predictor and cost predictor of PASTA directly from simulation and define heuristic controllers for the skills. For evaluation, we first generate desired target point cloud and then reset the dough to its initial shape and record its point cloud. We
7

---

Tup-aran
drugh
subgosi
Finzl target
push
CutReerrange
Orservation
Step
Tool changer
CRS
Robot setup
Stop
CRS-Twice
0,250
Shep
Shepp
Step
Step
Stop
Figure Real world setup and execution with planned subgoals. Our workspace consists of Franka robot, a top-down camera, a novel changer behind the robot that allows the robot automatically switch tools. each task, we show frames
executing skill overlaid with the decoded point cloud subgoal; we report the final performance in red and overlay the ground truth target in green in the final frame. Additionally, we include 3D view of the last generated subgoal to show the shape variations.
then plan using PASTA and then use the heuristic controllers to follow the plan. Finally, we report the normalized improvement EMD. We compare with the Flat3D method and also with human.
We evaluate three of the simulation tasks: CutRearrange, CRS, and CRS-Twice. For each task, evaluate the same four initial and target shapes for all methods and report the performance in Table 3. Figure 3 shows the keyframes from the execution of PASTA. We overlay the planned subgoals as well as the final goal. PASTA performs on par with human in the real world, highlighting the robustness of our planner and the advantage of using 3D representation for sim2real transfer.
Tisk (Horizon)
Mothod
CutRearrange
CRS
CRi-
Flat 3D
0.351 1 0.478
0.007 1. 0.429
PASTA (Ours)
0.836 0.029
0,854 + 0.016
0.795 + 0.035
Human
0.910 = 0.014
0.863 = 0.018
0.895 + 0.013
Table 3: Normalized improvement on real world tasks. Each entry shows the mean std performance over 4 runs. Flat 3D does not produce any meaningful plan for CRS, so we do not evaluate it on CRS-Twice.
5 Conclusions and Limitations
In this work, we propose a planning framework named PASTA that incorporates both spatial and temporal abstraction by planning with 3D latent set representation with attention structure. We demonstrate manipulation system in the real world that uses PASTA to plan with multiple tool-use skills to solve the challenging deformable object manipulation tasks, and we show that it significantly outperforms flat 3D representation, especially when generalizing to more complex tasks.
First, we only transfer the planner to the real world and use heuristic controllers instead of the policy trained in simulation. This due the sim2real gap caused by the differences in dough's physical parameters, table friction, and occlusions from the robot arm. Prior work [38] shows promising results in transferring closed-loop policy taking partial point clouds input, and future work can explore better sim2real methods, Second, our planner exhaustively searches overall skill combinations and attention structures and does not scale well to longer sequences with more skills. Future work can incorporate more efficient search algorithms or priors to prune the search space. Finally, we rely on an unsupervised clustering method for entity decomposition and a point cloud VAE for mapping an observation our latent representation. Future work can incorporate self-supervised methods for learning the decomposition. For further discussion the limitations and future work, please see Sec. 5 of the Appendix.
Limitations:

---

Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. IIS IIS-2046491, IIS- 1849154, NSF award under AWD00001520, LG Electronics, Carnegie Mellon University's ' GSA/Provost GuSH Grant funding, and Amazon Research Award.
References
[1] X. Ma, D. Hsu, and W. S. Lee. Learning latent graph dynamics for visual manipulation of deformable objects. In 2022 International Conference Robotics and Automation (ICRA), pages 8266-8277 8273. IEEE, 2022.
[2] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. arXiv preprint 19X3v:990300684, 2019.
[3] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[4] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference Robot Learning, 2021.
[5] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pages 373-385. PMIR, 2018.
[6] Y. Li, S. Li, V. Sitzmann, P. Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference Robot Learning, pages
PMLR, 2022.
[7] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In International Symposium of Robotics Research, 2022.
[8] X. Lin, Z. Huang, Y. Li, J. B. Tenenbaum, D. Held, and C. Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. ICLR, 2022.
[9] Nasiriany, H. Liu, and Y. Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In IEEE International Conference Robotics and Automation (ICRA), 2022.
[10] M. Dalal, D. Pathak, and R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances Neural Information Processing Systems, 34, 2021.
[11] C. R. Garrett, T. Lozano and L. P. Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 440-448 2020.
[12] M. A. Toussaint, K. R. Allen, K. A. Smith, and B. Tenenbaum. Differentiable physies and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation, 2018.
[13] R. Fikes and J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189 1971.
[14] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and
D. Wilkins. Pddl-the planning domain definition language. Technical Report CVC TR-98- 003/DCS 1R-1 1998.
[15] M. Toussaint. Logic-geometric programming: An optimization- approach combined task and motion planning. In Twenty-Fourd International Joint Conference Artificial Intelligence, 2015.

---

[16] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Somee:: Spatial object-centric representations for sequential manipulation. In Conference on Robot Learning, pages 148-157. PMLR, 2022.
[17] Liang, M. Sharma, A. LaGrassa, S. Vats. S. Saxena, and O. Kroemer. Search-based task planning with learned skill effect models for lifelong robotic manipulation. In 2022 IEEE International Conference on Robotics and Aulomation (ICRA). IEEE, 2022.
[18] E. Ugur and J. Piater. Bottom-up learning object categories, action effects and logical rules: From continuous manipulative exploration symbolic planning. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2627-2633, 2015. doi: 100//CAA. 2015.7139553.
[19] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. The International Journal of Robotics Research, 40(6-7):866-899)) 89466666994,922 2021.
[20] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.
[21] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference muchine learning, pages 2555-2565. PMLR, 2019.
[22] K. Fang, Zhu, A. Garg, Savarese, and Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. In Conference on Robot Learning, pages 42-52. PMLR, 2020.
[23] A. Simeonov, Y. Du, B. Kim, F. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. In 4th Conference on Robot Learning 2020). Massachusetts Institute of Technology, 2020.
[24] T. Pfaff, M. Fortunato, A. Sanchez-Gonnalez, and P. Battaglia. Learning mesh-based simulation with graph networks, In International Conference on Learning Representations, 2020.
[25] X. Ma. Hsu, and W. S. Lee, Learning latent graph dynamics for deformable object manipula- tion. arXiv preprint arXiv:2104. 12149, 2021.
[26] N. Heravi, A. Wahid, C. Lynch, P. Florence, T. Armstrong, J. Tompson, P. Sermanet, J. Bohg, and D. Dwibedi. Visuomotor control multi-object scenes using object-aware representations. arXiv preprint arXiv:2205.066: 00633,, 2022.
[27] C. Devin, P. Abbeel, T. Darrell, and S. Levine. Deep object-centric representations for gener- alizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7111-7118, IEEE, 2018
[28] D. Xu, R. Martin-Marth D.-A. Huang, Y. Zhu, S. Savarese, and L. F. Fei-Fei. Regression planning networks. Advances in Neural Information Processing Systems, 32, 2019.
[29] C. Schenck and D. Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 2629-26338 IEFE, 2017.
[30] G. Narasimhan, K. Zhang, B. Eisner, X. Lin, and D. Held. Transparent liquid segmentation for robotic pouring. In 2022 International Conference on Robotics and Automation (ICRA), 2022.
[31] Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg, Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 941 1-9418. IEEE, 2020.
10

---

[32] P. Mitrano, D. McConachie, and D. Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics, 6(54):eabd81 170, 2021.
[33] J. Maitin-Sheppard M. Cusumano-Townes J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308-2315 IEEE, 2010. URL ieee. ieee.org/eeeeeeeeeeee..e....e.ee poastracct/dccmeet///ttt
[34] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. Tanwani, N. Jamali, K. Yamane, S. Iba, and
K. Goldberg. VisuoSpatial Foresight for Physical Sequential Fabric Manipulation. Autonomous Robots (AURO) journal, 2021.
[35] Z. Huang. X. Lin, and Held. Mesh-based dynamics model with occlusion reasoning for cloth manipulation. Robotics: Science and Systems (RSS), 2022.
[36] T. Weng, S. Bajracharya, Y. Wang, K. Agrawal, and D. Held. Fabricflownet: Bimanual cloth manipulation with flow-based policy. In Conference on Robot Learning, 2021.
[37] H. Shi, H. Xu, Z. Huang, Y. Li, and Wu. Robocraft: Learning see, simulate, and shape elasto-plastic objects with graph networks. Robotics: Science and Systems (RSS), 2022.
[38] C. Qi, X. Lin, and D. Held. Learning closed-loop dough manipulation using a differentiable reset module. IEEE Robotics and Automation Letters, 7(4):9857-9864, 2022. doi: 10. 109/ 19..1992..
[39] N. Figueroa, A. L. P. Ureche, and A. Billard. Learning complex sequential tasks from demon- stration: A pizza dough rolling case study. In 2016 11th ACM/IEEE International Conference on Human-Robe Interaction (HRI), pages 61 1-112. Ieee, 2016.
[40] C. Matl and R. Bajesy. Deformable elasto- object shaping using elastic hand and basel-based reinforcement learning. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3955-3962. IEEF, 2021.
[41] E. Heiden, M. Macklin, Narang, D. Fox, A. Garg, and Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science and Systems, July 2021. doi: 10.::00.1660 1tt........0//..2222../// XXVI10067.
[42] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
[43] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525-11538 2020.
[44] G. Yang, Huang, Z. Hao, M.-Y. Liu, S. Belongie, and Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. ICCV, 2019.
[45] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[46] Y. Lee, E. S. Hu, and J. Lim. Ikea furniture assembly environment for honghoorionn complex manipulation tasks. In 202/ IEEE International Conference on Roboties and Automation (ICRA), pages 6343-6349, IEEE, 2021.
[47] Z. Huang, Hu, T. Du S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference Learning Representations, 2020.

---

[48] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[49] T. Séjourné, J. Feydy, F-X. Vialard, A. Trouvé, and G. Peyré. Sinkhorn divergences for unbalanced optimal transport. arXiv preprint arXiv: 2019.
[50] Y. Zhang. Library of behaviors and tools for robot manipulation. Master's thesis, Carnegie Mellon University, Pittsburgh, PA, August 2022.
12


Reviews:
[{"review_id": "xKXfKZPYa5X", "reviewer": "Reviewer_zTGC", "summary": "\nThis paper presents a planning framework for long-horizon manipulation over deformable objects. The approach authors present demonstrate good generalization ability to unseen number of objects and can incorporate quite a number of skills that involve deformable objects manipulation. The paper is important to the community in terms of scaling deformable object manipulation to long-horizon execution instead of always focusing on single-stroke, single-object manipulation. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The paper presents an approach that shows quite an impressive plan over deformable object manipulation.  The experimental results and the videos are quite impressive. I think the authors are tackling important questions in terms of deformable object manipulation.\n- The approach is designed in a way that it can learn skills in a simple simulator setting and use the skills in the unseen configurations which contain more objects.\n\nWeakness:\nI have quite a lot of comments on the paper writing. I think the paper writing needs improvements in terms of precise wording and related work citations:\n- Line 4: I suggest authors can write more explicitly why \"full-state\" or \"short-horizon\" assumptions prevent the use of deformable objects\n- Line 8: are authors using any other 3D observations? If not, \"such as\" is not precise here. \n- Line 22: Are the skills (temporal abstractions) for deciding which tools to use? or the abstractions of execution with skills? Make discrete decisions of which tool to use also sounds like a single-step decision.\n- Line 27: One issue I have is, why \"atoms\"? I think the community typically compares state abstractions with physical states or raw sensory representations. I don't think anyone does \"infinitesimally\" timesteps. The logic sounds ok but quite trivial. I think the authors can do much better in describing this part.\n- Line 28-35: I found this paragraph overclaiming. A lot of works use temporal abstractions in robot research and spatial abstractions. I think the authors are considering the context of deformable objects. Please change \"complex objects\" to \"deformable objects\" to make this paragraph sound specific and precise.\n- Line 40: What is \"nearly infinite\" degrees of freedom? Authors should provide a citation that strictly defines such an ambiguous term. Otherwise, I think “infinite” is sufficient\n- Line 58: Task and Motion Planning doesn't construct the models --- they assume/use the models given. \n- Line 72-78: missing citation -- such as regression planning network[1], which is grounds in visual observation but generalizes to long-horizon tasks.\n- Line 120: I am not very convinced by the term. It seems to describe a single abstraction that is both spatial and temporal. But from my understanding of the paper, spatial abstraction is on the 3D representation, and temporal abstraction is about the tool-use skills. I don't think this is an appropriate name for abstractions in the paper.\n- Line 168: Reward is quite an RL-oriented terminology, while the work focuses more on the planning work. I suggest using the term \"cost\" to better align with the planning literature. Also, that would align better with how pseudo code presents the approach.\n- And I think the specification of goals is not ideal. The goal is given in terms of point clouds, which means that the execution of the task is required before planning a sequence for such an execution. While the goal specification is not the main point of this paper, it hinders the proposed approach's significance. Specifying goals simply could make the paper to be more significant. I think the authors should also discuss this limitation in Section 5. In my opinion, the unsupervised learning part is quite minor compared to the limitation of goal specification.\n\n\n[1] Xu et al. Regression Planning Networks\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I would like to give weak accept to this submission based on its impressive results and the hard problem the authors try to solve. However, I think the paper presentation needs a boost in quality in terms of precise wording.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I hope most of my comments in the \"weakness\" part can be addressed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "rPhrK4Qp84t", "reviewer": "Reviewer_E6Aq", "summary": "This paper introduces a method that leverages spatial abstraction and temporal abstraction to facilitate planning for long-horizon deformable object manipulation. Temporal abstraction is achieved using primitive skills and spatial abstraction is achieved by independently representing each object in a scene. More specifically, the proposed model consists of 1) a Point Cloud VAE that encodes the point cloud of each object into a latent vector and can be used to generate imagined subgoals, 2) a set point cloud policy for each skill that operates on a set of object point clouds, 3) a set feasibility predictor that operates on latent codes of objects to predict skill feasibility, and 4) a set reward predictor that predicts distance to the goal. A planning algorithm uses the learned components to predict a sequence of primitive skills and the latent subgoals for the skills. The latent subgoals are decoded with the VAE decoder and used to condition the set point cloud policy.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Compared to the most-related prior work [7], this paper introduces set-based versions of the skill policy, feasibility predictor, and reward predictor using clever designs of the networks, objective functions, and planning algorithm. \n2. A nice property of the planning algorithm is that it can work with skills that change the number of objects in a scene.\n3. The attention mask is used to exclude irrelevant parts of the scene. The importance of this design is validated in Table 4 in the appendix.\n4. The results presented in Table 1 are very strong. The Flat3D and DiffSkill-Image baselines help to demonstrate the importance of spatial abstraction, which is the main contribution of this paper.\n5. It’s great to see that this method also generalizes well to the real robot using heuristic controllers for the skills.\n6. The discussion and ablation study of the feasibility predictor also provide a lot of insights on how to ensure its gradient can successfully guide the search for correct subgoals.\n\nWeakness:\n1. It would be great to see a discussion of the complexity of the algorithm in the main text. Since the planning algorithm performs nested optimization at 3 different levels. I am curious if the algorithm can easily scale to tasks that require chaining more than 6 skills (e.g., CRS-Third, CRS-Fourth, and etc) and more than 3 objects. Will the algorithm work if there are distractor objects in the scene?\n2. It would be interesting to test the generalization capability of the proposed method with more tasks. Can the method be evaluated on different combinations of the primitive skills (i.e., Lift, Spread, Gather, Transport, Cut, Rearrange).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper combines spatial and temporal abstraction to achieve long-horizon deformable manipulation. The proposed planning framework and neural network modules can deal with variable numbers of objects and compose primitive skills to manipulate more objects in test time. The experiment shows that this object-centric scene representation is more effective than existing methods that do not use spatial abstraction. The real robot experiment also validates the proposed method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Besides the questions I raised above, I have the following comments:\n1. In Line 91, $T$ is used to denote the number of actions between the initial and goal. $T$ is used later to represent the skill horizon. Since multiple skills are executed between the initial step and the goal step, maybe a different symbol should be used.\n2. In Line 92, $i$ is used to index time but it is later used to index objects.\n3. Line 117 is confusing since it states that the point cloud of each object is encoded in a set of latent vectors. \n4. Line 130 states that the set point cloud policy takes in the observed point cloud $P_{obs}$. Doesn’t the policy network take in a set of object point clouds {$P_1,...,P_C$} instead? I recommend the authors more clearly differentiate point cloud sets and the point cloud of the whole scene in the method section.\n5. Line 175, why is the MLP trained to predict Chamfer distance between latent vectors?\n6. Why did PASTA achieve better performance on CutRearrangeSpread(CRS) than CutRearrange? Does this have to do with the different training for the two tasks?\n7. Looking at Step 3 of CRS-Twice in Figure 3, the predicted subgoal does not match the actual point cloud of the left object but this task is still completed successfully. Can the authors provide more insight into the robustness of the algorithm to errors in subgoal prediction and whether the policy networks are using the predicted subgoals? \n"}}, {"review_id": "njGceQ2lO3_", "reviewer": "Reviewer_jQsA", "summary": "This work is about long-horizon elastoplastic object manipulation, including multiple tools. This paper emphasizes that their method incorporates spatial abstraction and temporal abstraction. Here, spatial abstraction is about decomposing a dense scene into a set of objects and representing them with embeddings, and temporal abstraction is about high-level reasoning over different skills. PASTA executes a trajectory by solving a two-step optimization problem where the first step finds an optimal sequence of skills and the second step finds an optimal sub-goal for each skill.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Advantages**\n- Suggest planning with tool changes and learning individual policies for each tool.\n- Simulation and Real-world experiments using different types of elastoplastic material(sand and real dough).\n\n**Weaknesses**\n- One of the main limitations of this paper is that their ‘point cloud policy’ or the action generator is not evaluated in or transferred to real-world experiments. For the real experiment, they utilized heuristic controllers for the skills.\n- PASTA does not allow feedback between each subgoal in a sequence (k_1, …, k_H). Planning in the long-horizon is a great direction; however, not updating subgoals during the long-horizon manipulation makes the method susceptible to disturbances.\nRequires to specify the number of steps during tests. Performance efficiency and accuracy depending on user input can also be the limitation of this work. \n- PASTA relies on manual pre-designed skill definitions (M_k_h, N_k_h). \n- Lack of clarifications/typos in some texts \n    - Line 86-88: I am not sure how ‘using 3D set representation extracted from point clouds’ can directly be linked to generalizing compositionally to tasks with \"more objects and longer horizon”.\n    - Line 130 : can you clarify the length of action from \\pi? The output actions per each skill should be greater than 1, especially for spreading, which makes the robot move the roller back and forth. \n    - Line 138 : definition of N_o and N_g\n    -Typo in line 116: “We can the encode~ “", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Elasto-plastic objects are generally hard to manipulate with robotic fingers or grippers, thus requiring tool use. This study is powerful in that it allows deformable object manipulation with multi-tools. However, I am hesitant to recommend this paper unless the authors address the major limitations. As discussed in 'issues', the major concerns are 1) partial sim2real transfer and potentially misleading evaluations and 2) lack of justifications in model choice/design (2-4 bullet points in issues). Other than that, thorough real-world analysis and experiment are great, and the paper overall was enjoyable to read.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Doughs in real life are much more challenging to handle because it sticks to the desk and the tool. Also, dough's material property is time-variant. (Get firmer and less sticky as it dries). This seems to be a great huddle for PASTA's sim2real transfer. Including the authors' insight on integrating these uncertainties with an action generator will make this study much stronger. \n- Justification of using tool point cloud: Why use the tool 'pointcloud' for Point Cloud Policy instead of the estimated pose and the type of the tool? For example, the tool can be illustrated with 6 numbers (x,y,z,roll,pitch,yaw) where the type of the tool can be expressed in one-hot [0 ... 1 0 ..]. I wonder how robust pointcloud-based tool input regarding the tool occlusion caused by the robot arm/gripper and dough. \n- Why do you think decoupling subgoal and skill sequence for optimization is a great choice? Why not optimize them simultaneously?\nWhy is it better to use a reward 'predictor'? (Line 173) Can we convert the decode the latent codes and directly get the reward? For example, you can explicitly use grid search to get the similarity score.\n- I feel that the author's heuristic controllers in the real world might conceal the sub-goal and action error. In detail, in 'Fig.3. CutRearrange' step 2 (top sand) shows that PASTA shows some error at 1) setting accurate subgoal and 2) moving the dough to the target place. Ironically, the errors devoted to increasing the 'Final' accuracies, which I think is odd. Can the authors explain this?\n\n**Writing**\nExplicitly making a section describing the training process and the entire loss formulation will help readers to follow the paper.\nHow much train and test data? What is their distribution?"}}, {"review_id": "i6HP5eUQZPX", "reviewer": "Reviewer_ZcC8", "summary": "This paper presents a learning pipeline that combines spatial and temporal representation to conduct long-horizon planning for deformable object manipulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is very well written and easy to follow. The framework is extensively tested with simulation and real robot experiments. The website and appendix provide complementary information which helps the reviewer to better understand the proposed approach. \n\nWeaknesses:\n\nThe reviewer found the Limitations paragraph unsatisfactory. The first limitation talks about the limited generalization capability of learned parameters to different tasks which are acceptable. The second limitation on the sim-to-real gaps seems to be more like a statement rather than pointing out the limitation.\n\nMinor:\n- P.4. Line 99, one of the K skills, K is not defined.\n- P.4. Line 122-123 “a single-stage task with a single tool and can be just for a single object”, maybe it is better just “a single-stage task with a single tool for a single object” \n- P.5. Alg 1  Decode subgoal from $U^h$ should be $U_h$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "A decent paper with extensive validation on simulation and real robot experiments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Limitations could be elaborate on a bit more. \n- The paper considers 3 skills, respectively push, cut, and spread, and the longest task requires applying 6 skills in sequence. This makes the possible combination of actions #3 * 2^5# = 96 variations. This is a rather small search space to find the correct sequence, the reviewer wonders if the authors could comment on how well the method scale to larger search space for task sequencing."}}]

Author rebuttal:

Decision: Accept (poster)