RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools
Authors: Haochen Shi
Huazhe Xu
Samuel Clarke
Yunzhu Li
Jiajun Wu
Venue: CoRL 2023 Oral (2023)
Keywords: Deformable Object Manipulation
Long-horizon Planning
Model Learning
Tool Usage

Abstract: Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies. Extensive evaluations show that RoboCook substantially outperforms state-of-the-art approaches, exhibits robustness against severe external disturbances, and demonstrates adaptability to different materials.

RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools
Haochen Shi!"
Huazhe Xu
Samuel Clarke!
Yunzhu Lil.2
Jiajun Wu
'Stanford University
UICC
*Equal contribution
https: http///hhiiiiii
www.ggitthh...ooo itoo://coooooooo robocook
Abstract: Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, Robo- Cook, which perceives, models, and manipulates elasto-plastic objects with vari- ous tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classifi- cation with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipula- tion tasks, such as making dumplings and alphabet letter cookies. Extensive evalu- ations show that RoboCook substantially outperforms state-of-the-ar approaches, exhibits robustness against severe external disturbances, and demonstrates adapt- ability to different materials.
Keywords: Deformable Object Manipulation, Long-horizon Planning, Model Learning, Tool Usage
Introduction
Think about all the steps and tools robot would need to use make dumpling from lump of dough. This scenario contains three fundamental research problems in robotics: deformable object manipulation [1, 2, 3, 4], long-horizon planning [5, 6, 7, 8), and tool usage [9, 10, 11, 12]. The task poses significant challenges to the robot, because it involves decisions at both discrete (c.g., which tool to use) and continuous levels (e.g., motion planning conditioned on the selected tool).
To address these challenges, we propose RoboCook, framework that perceives, models, and ma- nipulates elasto-plastic objects for long-horizon tasks like making dumplings and alphabet letter cookies. RoboCook introduces three technical innovations. First, we apply data-driven approach with Graph Neural Network (GNN) [13, 14, 15) to learn highly complex interactions between the soft object and various tools purely from visual observations. Second, we combine a PointNet- based [16, 17] tool classification module with learned dynamics models to determine the most appropriate tool to use at the current task stage. Third, we use self-supervieed policy trained with synthetic data generated by our learned dynamics model for gripping. rolling, and pressing to improve performance and speed, and hand-coded policies for other skills.
We carry out comprehensive evaluations to show RoboCook's 's effectiveness, robustness, and gener- alizability. Figure shows a typical successful trial of making a dumpling. To showcase robustness, we apply external perturbations during real-time execution, and RoboCook still succeeds in making a dumpling. To demonstrate generalizability, we test RoboCook to make alphabet letter cookies and RoboCook outperforms four strong baselines by significant margin. RoboCook can also generalize various materials by making particular shape different materials without retraining.
Now at Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Initial state
Knife
Twa--plaee symmetric gripper
Square press
Large raller
Circle cutter
Pusher
Skin spatula
Filling
Hook
Final
Figure 1: Making dumplings. RoboCook makes a dumpling from a piece of dough in nine steps: appropriate volume, (2) pinches the dough and regularizes the shape, (3) presses to flatten the dough, (4) rolls to flatten the dough further, (5) cuts circular dumpling skin, (6) removes the excess dough, (7) picks and places the skin onto the mold, (8) adds the filling, and (9) closes and opens the mold. The black arrows denote the moving direction.
The robot (1) the dough
2 Related Work
Long-horizon deformable object manipulation. Real-world deformable object manipulation is challenging task [18, 19, 20]. The innate high DoF's partial observability, and non-linear local in- teractions make deformable objects hard to represent, model, and manipulate [21, 22, 23]. Conven- tional approaches in deformable object manipulation choose model-based methods [24, 25, 26, 27] or adaptive methods [28, 29, 30), yielding complex system identification problems. For example, Li al. [27] proposes differentiable simulator as the surrogate of real-world elasto-plastic objects. However, these simulators are based on approximate modeling techniques, resulting in noticeable sim-to-real gap that hinders their application to real-world scenarios. There have been efforts toward learning to manipulate from expert demonstrations [31, 32, 33]. This paradigm is used in manipu- lating liquid [11], sand [29]. and dough [34]. Despite these successes, obtaining the demonstration expensive and sometimes prohibitive. Another recent trend is to learn a dynamics model from high-dimensional sensory data directly for downstream manipulation tasks [18, 35, 36, 37, 15, 38). While these approaches show impressive results, they only consider short-horizon tasks using just tool. In contrast, long-horizon tasks like dumpling-making require reactive planner to under- stand the Long-term physical effect of different tools to make the most effective discrete (e.g.. which tool use) and continuous (e.g., the action parameters) action decisions.
Tool usage. Tool usage widely studied in cognitive science and robotics research [39, 40]. To study the process human evolution, many researchers endorse tool usage as a main benchmark evaluate the intelligence of living primates with respect to that of extinct hominids [41, 42, 43]. To equip robots with the same capability of tool usage humans, prior works focus teaching the robot the representation and semantics of tools objects that potentially function like tools for downstream policy learning [11, 12]. To perform complicated long-horizon tasks composed of several subtasks, prior works also use human demonstrations to learn hierarchical policy net- work [44]. In this work, we will use real-world robot random-play data, which is cheaper to acquire than human demonstration data. We train the robot on these self-exploratory trials to understand the physical interactions between different tools and deformable objects.
3 Method
The RoboCook framework has three major components: perception, dynamics, and closed-loop control. We first use an effective sampling scheme and intuitive tool representations for particle-

---

Perception
Dynamics
Initial State
dg
{
d:
dg
dg
4
GNN
GNN
5
GNN
GNN
GNN
PRED
Gripping
Top
Raw point
cloud cloud
Pers.
Side
PRED
Pressing
Side
GT.
(a)
(b)
Top
Pers.
PRED
Dough Tool Outliers
Rolling
Pers.
GT
(d)
(e)
Top
Side
Figure 2: Perception and dynamics of RoboCook. (A) The input to the perception module is point cloud the robot's workspace captured by four RGB-D cameras. From raw point cloud, (a) crop the region of interest, (b) extract the dough point cloud, (c) reconstruct watertight mesh (d) use the Signed Distance Function (SDF) to sample points inside the mesh,
(c) remove points within the tools' SDF, and (f) sample 300 surface points. (B) We process videos of each tool manipulating the dough into a particle trajectory dataset to train our GNN-based dynamics model. The model can accurately predict long-horizon state changes of the dough in gripping, pressing, and rolling tasks. On the left are the initial states' perspective, top, and side views, and on the right is comparison of model predictions and ground truth states.
based scene representation. Second, we train Graph Neural Networks (GNNs) as the dynamics model from the processed video dataset accurately predict dough states during manipulation. Last, tool classifier selects the best tool for each substage long-horizon task and self-supervised policy network performs closed-loop control.
3.1 Perception
The perception module aims to sample particles sparsely and uniformly for the downstream dy- namics model. This task is challenging because of dough occlusions from the robot and tool and self-occlusions from the irregular and concave shape of the dough.
We merge point clouds from four calibrated RGB-D cameras and perform color segmentation to extract the dough point cloud. Then apply either Poisson surface reconstruction [45] or alpha- shape surface reconstruction [46] to reconstruct watertight surface of the dough, depending occlusion levels. In heavy occlusion cases, alpha-shape surface reconstruction is usually worse at capturing concavities than Poisson surface reconstruction. However, we use it to secure a complete and singular mesh. With few or no occlusions, we combine Poisson surface reconstruction and the MeshFix algorithm [47] to generate watertight mesh. We use the watertight mesh's Signed Distance Function (SDF) to sample points inside it randomly. To compensate for details lost during surface reconstruction, apply voxel-grid filter to reconstruct concavities by removing the points above the original dough point cloud. We also remove the noisy points penetrating the tool using the mesh's 's SDF. We compute the tool mesh's SDF from the robot's end-effector pose (recorded during data collection) and the ground-truth tool mesh. We then perform alpha-shape surface reconstruction and use Poisson disk sampling [48] to sample 300 points uniformly on the surface, capturing more details of the dough with a fixed particle number.
For different tools, we uniformly sample particles the surface of their ground truth mesh to reflect their geometric features. The complete scene representation for the downstream dynamics model

---

concatenates dough and tool particles. Section 6.2.1 and 6.2.2 of supplementary materials describe more data collection and preprocessing details.
3.2 Dynamics Model
Our ENNN-asee dynamics model predicts future states of dough based on the current state and actions of the tools with only 20 minutes of real-world data per tool. The rigid and non-rigid motions each predicted by multi-layer perceptron (MLP) and added together as the predicted motion by the GNN.
The graph of sampled particles at each time step is represented as 8 = (O,, Ec) with O2 as vertices, and & as edges. For each particle, og, = (x,,, c,,)), where X is the particle position at time t, and cf is the particle's ' attributes time t, including the particle normal and group information (i.e., belongs the dough the tool). The edge between pair of particles is denoted e (ux, *k), where 1 S Uk, Up < Or] are the receiver particle index and sender particle index respectively, and is the edge index. Section 6.3.1 of supplementary materials provides more details on graph building.
Following the previous work [18], we use a weighted function Chamfer Distance (CD) [49] and Earth Mover's Distance (EMD) [50] as the loss function to train the dynamics model:
L(O,, O.) W1 LCD(O,, O.) + w2 LEMD O.),
where O is the real observation, and O the predicted state. We select wi = 0.5 and w2 = 0.5 based on the experiment results. To be more specific, the CD between O2 O C R calculated by
Lo,, E min
||x
y1l2 + Σ
min ||x - y x- XEO,
yEO,
XEO,
Yεδ.
The EMD matches distributions of point clouds by finding a bijection μ O Ö2 such that
LEMD (O,, O,) = min Σ
(xx)(x)]12.
33)
m:O.- xEO.
We train the model to predict multiple time steps forward to regularize training and stabilize long- horizon future predictions. The training loss is calculated as a sum of distances between predictions and ground-truth states:
Luain
C(Orte, Octe).
(4)
1=0
where the dynamies model takes Õzts-1 as the input to predict Õ445 when 0. The model performs better inference time by predicting a slightly longer horizon during training time. Em- pirically, discover that s 2 good enough for inference-time prediction accuracy, and larger does not give much gain the accuracy. Note that although predict 15 steps during inference time, which is long-horizon prediction. This highlights the in- ductive bias of our GNN-based dynamics model - we only need to train on short-horizon predictions to generalize to a much longer-horizon prediction during inference. Another motivation to keep small is to increase the training speed. More implementation details on model training can found in Section 6.3.2 of supplementary materials.
2 during training,
3.3 Closed-Loop Control
To solve long-horinn horizoorzon manipulation tasks involving multiple tools, the robot must determinc (1) the best tool to use at each stage and (2) the optimal trajectory for the selected tool.
To answer the first question, we implement tool classification module based on PointNet++ [17] that takes a concatenation of the input and output states of the dough as input and outputs probabil- ities of the 15 tools achieving the target state. We extract the features of the input and output point clouds separately with PointNet++ classification layer. Then feed the concatenated features to an MLP with a SoftMax activation layer to output probabilities.

---

A. Closed-Loop Control
Current
Target
Large roller
Actions
Updated
Circle press
Circle punch
Classifer Network
Policy Network
Knife
Circle cutter
Hook
Press and roll
Visual feedbackk
B. Self-uupervised policy learning
Gripping
Pressing
Rolling
Tool
Randomly sampled action
Su
S.
GNN
(r,8,d)
(x,y,2,8)
(x,
GNN
GNN 1
GNN I
Policy
a(
Network
gradient
Action space
Action space
Action space
Figure 3: Planning of RoboCook. (A) PointNet-based classifier network identifies appropriate tools based on the current observation and the target dough configuration. The self-supervised policy network takes the tool class, the current observation, and the target dough configuration as inputs and outputs the manipulation actions. The framework closes the control loop with visual feedback. (B) We show the policy network architecture, the parametrized action spaces of gripping, pressing, and rolling, and how we generate the synthetic datasets to train the policy network.
The three tools with the highest probabilities are selected as candidates, and their optimal actions are planned. The robot selects the tool yielding the final state closest to the target and executes the planned actions. We let the module output three tool candidates instead of only one to increase our pipeline's error tolerance to the tool classifier.
To address the second question, specify and parameterize the action space of the 15 tools based on human priors. Then we classify the tools into few categories based on [A], the dimension of their action space. Section 6.4.1 of supplementary materials provides more details on the specification and justification of tool action spaces.
For grippers, rollers, presses, and punches, we collect data in the real world by random exploration in the bounded action space and learn their interaction model with the dough using GNN. Then generate synthetic dataset with the dynamics model, train self-supervised policy network to obtain the optimal policy, and execute closed-loop control with visual feedback. We use straightfor- ward planners for other tools based on human priors. More implementation details can be found in Section 6.4.1 of supplementary materials.
To generate the synthetic dataset, we reuse the initial dough states acquired during data collection and randomly sample actions in the parameterized action space of each tool. As shown in Fig- ure 3(A), the dough states before and after applying actions the input, and the action parameters are the labels. We design a self-supervised policy network based on PointNet++. Given the continu- ous solution space of the action parameters, we formulate a multi-bin classification problem inspired by previous works on 3D bounding box estimation [51, 52]. First, we concatenate input and output point clouds, labeling them or for distinction. Next, we extract features using PointNet++ layer. Finally, we feed the feature vector into separate classification and regression heads, both be- ing MLPs with identical structures. The learned policy drastically improves the planning efficiency, and the complete pipeline only takes around 10 seconds of plamning time make dumpling. More implementation details are described in Section 6.4.2 of supplementary materials.
During closed-loop control, cameras capture the workspace point cloud, which the perception mod- ule processes to obtain a clean and sparse point cloud of the dough. Our method takes the current observation and final target as input and returns the best tool and optimal actions, as shown in Fig- ure 3(B). Then the robot picks up the selected tool, manipulates the dough, lifts its hand to avoid occluding the camera views, and plans the next move with visual feedback.
5

---

Figure 4: RoboCook hardware and setup. Left: Robot's 's tabletop workspace with zy² coordinates top-left. Dashed white circles: four RGB-D cameras mounted at four corners the table. Red square: dough location and manipulation area. Dashed white square: tool racks. Right: 15 tools: (1) large roller, (2) circle press, (3) circle punch, (4) square press, (5) square punch, (6) small roller, (7) knife/pusher, (8) circle cutter, (9) hook, (10) dumpling mold, (11) two-rod symmetric gripper, (12) asymmetric gripper, (13) two-plane symmetric gripper, (14) skin spatula, (15) filling spatula. Tools are 3D-printed, representing common dough manipulation tools. Section 6.5.2 of supplementary materials discusses the design principles of these 3D-printed tools.
4 Experiments
In this study, we design and implement a hardware setup for soft body manipulation tasks, allowing easy selection and interchange of 15 tools, as shown in Figure We demonstrate the RoboCook framework's 's effectiveness in long-horizon soft object manipulation tasks that require the use of mul- tiple tools, such making dumpling from randomly shaped dough and shaping alphabet letter cookies to compose the word 'RoboCook." Additionally, RoboCook can complete these tasks under extensive human interference, including significant changes to the shape and volume, as shown in the second supplementary video, demonstrating its robustness to external perturbations. We discuss the experiment setup in Section 6.5 of supplementary materials.
4.1 Making Dumplings
The humpling-making task to manipulate a piece of dough and the given filling into dumpling shape. The main challenge lies in choosing appropriate tools and planning effective action trajecto- rics. We consider the task successful if the dumpling skin is thin enough and completely covers the filling. Hompling-making is a highly challenging task-eve a single error might break the entire pipeline. Our method reliably accomplishes the task as shown in Figure 1. We show a comparison with manipulation results of human subjects in Section 6.6 of supplementary materials.
RoboCook also demonstrates robustness against external perturbations during real-time execution. At each stage of dumpling making, human disturbs the robot by deforming the dough, adding additional dough, or even replacing the dough with a completely different piece to deviate from the trajectory of the robot's 's original plan. For example, at the rolling stage, the human folds the flattened dough into a bulky shape. The robot decides to roll again to get a flatter dumpling skin. In a more challenging case, the human replaces the round dumpling skin with a completely different dough in a highly irregular shape. After this perturbation, the robot puts down the roller, picks up the knife to start again from the beginning, and successfully makes dumpling. The complete process is shown in the second supplementary video. We also show quantitative evaluation of the tool classification network in Section 6.7 of supplementary materials.

---

Initial state
Step
Step
Ours
Outline
RoboCraft
CEM+GNN
CEM+MPM
RL+GNN
Figure 5: Making alphabetical letter cookies. We list R, O. B, C. and shaping steps in Columns to 4. Column 5 manually highlights the contour of the alphabetical letters. Columns 6 through compare our self-supervised learned policy with three model-based planning baselines and one RL. baseline. Our method can shape the dough closer to the target than all four baseline methods.
Methods
CD 1
EMD 1
CD of normal
Human evaluation
Planning time (8) J
Ours
0.0062
0.0007
0.0042 + 0.0006
0.1933 + 0.0345
0.90 + 0.11
9.3 + 1,5
RoboCraft
0.0066 + 0.0005
0.0044 = 0.0006
0.2011 = 0.0329
0.54 + 0.43
613.7 + 202.7
CEM+GNN
0.0066 0.0007
0.0045
0.0008
0.2043 1 0.0431
0.52 1 0.41
756.0 1 234.5
CEM+MPM
0.0070 0.0007
0.0046
0.0006
0.1965 + 0.0265
0.48 0.35
1486.7 512.8
RL+GNN
0.0077 0.0007
0.0064 = 0.0009
0.2041 + 0.0414
0.17 0.09
1867.9 190.3
Table 1: Quantitative evaluations. We use CD and EMD between the point clouds and the CD between the surface normals to evaluate the results. further profile how long these methods take to plan actions. Our method outperforms all baseline methods in these metrics by large margin.
4.2 Making Alphabet Letter Cookies
The RoboCook framework demonstrates effectiveness and robustness highly complicated ma- nipulation tasks such as dumpling mompling-making.. This section explores its generalization ability in tasks requiring precise actions such as shaping alphabet letter cookies [18] without additional training.
Figure 5 shows that the RoboCook framework can accurately shape letters R, O, B, C, and K to compose the word 'RoboCook.' For R, the robot uses the two-rod symmetric gripper for cavities and the circle punch for the hole. For O, the tool classifier selects a two-plane symmetric gripper to pinch the square into circle and the circle punch for the hole. Our method identifies the asymmetric gripper as suitable for and locates accurate positions for the circle punch to press twice. Shaping C is more challenging due to the large distance between the initial and target shapes, but method successfully plans gripping positions using closed-loop visual feedback. The robot combines the two-rod symmetric and asymmetric gripper for K to create cavities.
We compare our method with four strong baselines: (1) limited-memoot BFGS [53] with GNN- based dynamics model (RoboCraft) [18]; (2) cross-entropy method (CEM) [54] with GNN-based dynamics model; (3) CEM with Material Point Method (MPM) simulator [55]; and (4) Reinforce- ment Learning with GNN-based dynamics model. Qualitative results shown in Figure 5. We
include detailed analysis of our results in Section 6.10.

---

Material
Initial state
Manipulation Steps
Final State (K)
Real Dough
Play- Dooh
Air Dry Clay
Model Foam
Figure 6: Generalizing to different materials. We showcase the dynamics model's capability to generalize to various materials by shaping a K without retraining.
Table 1 evaluates the results using Chamfer Distance (CD) [49]. Earth Mover's Distance (EMD) [50] and CD between top surface normals. However, we recognize a discrepancy between how these metrics measure the results and how humans perceive them these metrics are prone to local noises while humans are good at capturing the holistic features of the dough. Therefore, we show the pre- diction accuracy of 100 human subjects on recognizing the letters for these five methods in Table Another highlight of our method is its speed. Since the policy network only needs one forward prediction output the actions for each tool, it is significantly faster than other methods that rely on forwarding the dynamics models many times. Section 6.8 and 6.9 of supplementary materials discuss more details about the human study and the baseline implementations.
We show that the dynamics model can generalize to various materials by shaping a K without re- training Figure 6. We test on Play-Doh, Air Dry Clay, and Model Foam, each displaying notably different dynamics compared to our original dough made of flour, water, and salt. The dynamics model is trained solely on interaction data with the real dough.
5 Conclusion and Limitations
RoboCook demonstrates its effectiveness, robustness, and generalizability in elasto-plasti object manipulation with a general-purpose robotic arm and everyday tools. The main contributions of RoboCook include (1) tool-aware GNNs to model long-horizon soft body dynamics accurately and efficiently. (2) a tool selection module combined with dynamics models to learn tool functions through self-exploratory trials, and (3) a self-supervised policy learning framework improve the performance and speed significantly. RoboCook pioneers solutions for tool usage and long-horizon elasto-plastic object manipulation in building generic cooking robot.
One limitation of RoboCook is the occasional failure of dough sticking to the tool. A solution is to design an automatic error correction system. RoboCook also relies on human priors of tool action spaces to simplify planning. But these simplifications do not constrain generalization as they can be easily specified for new tools. Section 6.4.1 provides more justifications for this. Another limitation that humans define the subgoals. Higher-level temporal abstraction and task-level planning are required to get rid of them. Finally, RoboCook requires additional topology estimation to apply to cables and cloths [56], which is beyond the focus of this work.

---

Acknowledgments
This work was in part supported by AFOSR YIP FA9550-23-1-0127, ONR MURI N00014-22-1- 2740, the Toyota Research Institute (TRI), the Stanford Institute for Human-Centered Al (HAD) JPMC, and Analog Devices.
References
[1] J. Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734-743. PMLR, 2018.
[2] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference Robot Learning, pages 432-448, PMLR, 2021.
[3] Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, et al. Challenges and outlook in robotic manipulation of deformable objects. IEEE Robotics & Automation Magazine, 29(3):67-77, 2022.
[4] H. Yin, A. Varava, and D. Kragic. Modeling, learning, perception, and control methods for deformable object manipulation. Science Roboties, 6(54))eabd8803, 2021.
[5] V. N. Hartmann, Orthey, Driess, O. Oguz, and M. Toussaint, Long-horizon multi-robot rearrangement planning for construction assembly, IEEE Transactions on Robotics, 2022.
[6] Nair and C. Finn. Hicrarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. International Conference on Learning Representations, 2020.
[7] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. In Conference on Robot Learning, pages 471-484. PMLR, 2021.
[8] A. Simeonov, Y. Du, B. Kim, F. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. In Conference on Robot Learning, pages 1582-1601. PMLR, 2021.
[9] A. Billard and D. Kragic. Trends and challenges in robot manipulation. Science, 364(6446): 44 2019.
[10] N. Yamanobe, W. Wan, I. G. Ramirez-Alpizar, D. Petit, T. Tsuji, S. Akizuki, M. Hashimoto, K. Nagata, and K. Harada. A brief review of affordance in robotic manipulation research. Advanced Robotics, 31 31(19-20):1086-11 2017.
[11] D. Seita, Y. Wang. S. J. Shetty, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In 6th Annual Conference on Robot Learning. 2022.
[12] A. Xie, F. Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects tools with visual foresight. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgau Germany, June 2019. doi: 1tt.:0..5507/RSS.2000 201............000....
[13] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61 2008
[14] Y. Li. J. Wu, Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference Learning Representations, 2018.
[15] Y. Li, J. Wu, J.-Y. Zhu, B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference Robotics and Automation (ICRA), pages 1205-1211. IEEE, 2019.

---

[16] C. R. QI, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[17] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space, Advances neural information processing systems, 30, 2017.
[18] H. Shi, H. Xu, Z. Huang. Y. Li, and Wu. RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022.
[19] C. Matl and R. Bajesy. Deformable elasto-plastic object shaping using elastic hand and modcl-based reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 3955-3962 IEEE, 2021.
[20] X. Lin, C. Qi, Y. Zhang. Z. Huang. K. Fragkiadaki, Y. Li, C. Gan, and D. Held. Planning with
abstraction from point clouds for deformable object manipulation. In 6th Annual Conference on Robot Learning, 2022.
[21] D. Driess, Z. Huang, Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. Conference on Robot Learning, pages 1755- 1768. PMLR, 2023.
[22] C. Chi, B. Burchfiel, Cousineau, Feng, and Song. Iterative Residual Policy for Goal- Conditioned Dynamic Manipulation of Deformable Objects. In Proceedings of Robotics: Sci- ence and Systems, New York City, NY, USA, June 2022. doi: 10d.100.1607//SS.2000000 202./XXIS..22...111......
[23] H. Ha and S. Song. Flingbot:
The unreasonable effecttivnness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pages 24-33. PMLR, 2022.
[24] Z. Huang, Y. Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, 2020.
[25] X. Lin, Z. Huang. Li, J. B. Tenenbaum, D. Held, and C. Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. In International Conference on Learning Representations, 2021.
[26] A.-M. Cretu, P. Payeur, and E. M. Petriu. Soft object deformation monitoring and learning for model-based robotic hand manipulation. IEEE Transactions on Systems, Man, and Cybernet- ics, Part B (Cybernetics), 42(3):740-755: 2011.
[27] S. Li. Z. Huang. T. Du, H. Su, J. B. Tenenbaum, and C. Gan. Contact points discovery for soft-body manipulations with differentiable physics. In International Conference Learning Representations, 2021.
[28] D. Navarro Alaaroo,, H. M. Yip, Z. Wang, Y.H.. Liu. F. Zhong. T. Zhang, and P. Li. Automatic 3-d manipulation of soft objects by robotic arms with an adaptive deformation model. IEEE Transactions on Robotics, 32(2):4294411, 2016.
[29] A. Cherubini, Ortenzi, A. Cosgun, R. Lee, and P. Corke. Model-free vision-based shaping of deformable plastic materials, The International Journal of Robotics Research, 1739 1759, 2020.
[30] K. Yoshimoto, M. Higashimori, K. Tadakuma, and M. Kaneko. Active outline shaping of rheological object based plastic deformation distribution. In 2011 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1386-1391. IEEE, 2011.
10

---

[31] B. Balaguer and S. Carpin. Combining imitation and reinforcement learning to fold deformable planar objects. In 201 JEEERSS International Conference on Intelligent Robots and Systems, pages 1405-1412. IEEE, 2011.
[32] Nadon, A. J. Valnncia, and P. Payeur. Multi-modal sensing and robotic manipulation of non-rigid objects: A survey. Robotics, 7(4):74, 2018.
[33] B. Jia, Z. Pan, Z. Hu, J. Pan, and D. Manocha. Cloth manipulation using random- undom-foreet based imitation learning. IEEE Robotics and Automation Letters, 4(2):2086 2093, 2019.
[34] N. Figueroa, A. L. P. Ureche, and A. Billard. Learning complex sequential tasks from demon- stration: A pizza dough rolling case study. 2016 1 11th ACM/IEEE International Conference on Stuman-Robot Interaction (HRI), pages 61 1 1-612. leee, 2016.
[35] Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[36] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.0034/1 2016.
[37] A. Sanzaez-Gonzalez, Heess, Springenberg. J. Merel, M. Riedmiller, R. Hadsell, and Battaglia. Graph networks as learnable physics engines for inference and control. In Inter- national Conference on Machine Learning, pages 4470 4479. PMLR, 2018.
[38] T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Lozeno-Pérez, and L. P. Kaelbling. Plan- ning with learned object importance in large problem instances using graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 1192--1177,, 2021.
[39] C. P. Van Schaik and G. Pradhan. model for tool-use traditions in primates: implications for the coevolution of culture and cognition. Journal of Human Evolution, 44(6):645-664, 2003.
[40] R. Holladay, T. Lozano-Pérez, and A. Rodriguez. Force-and-motion constrained planning for tool use. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7409-7416. IEEE, 2019.
[41] T. Parker and K. R. Gibson. Object manipulation, tool use and sensorimotor intelligence feeding adaptations in cebus monkeys and great apes. Journal of Human Evolution, 6(7): 623-641, 1977.
[42] T. Ingold. Tool-use, sociality and intelligence. Tools, language and cognition in human evolu- tion, 429(45):449-72, 1993.
[43] Matsuzawa. Primate foundations of human intelligence: a view of tool use nonhuman primates and fossil hominids. In Primate origins of human cognition and behavior, pages 3-25. Springer, 2008
[44] J. Liang, B. Wen, K. Bekris, and A. Boularias. Learning sensorimotor primitives of sequential manipulation tasks from visual demonstrations. In 2022 International Conference on Robotics and Automation (ICRA), pages 8591-85971 IEEE, 2022.
[45] M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, 2006.
[46] H. Edelsbrunner and E. P. Mücke. Three-dimensional alpha shapes. ACM Transactions Graphics (TOG), 13(1):43-72, 1994.

---

[47] M. Attene, A lightweight approach to repairing digitized polygon meshes. The visual com- puter, 26111):199-1446, 2010.
[48] C. Yuksel. Sample elimination for generating poisson disk sample sets. Computer Graphics Forum, 34(2):25-32, 2015.
[49] H. Fan, Su, and I.. Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[50] Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
[51] A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings the IEEE conference on Computer Vision and Pattern Recognition, pages 7074-7082, 2017.
[52] A. Kundu, Y. Li, and J. M. Rehg. 3d-remn: Instance-level 3d object reconstruction via render- and-compare. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pages 3559-3568, 2018.
[53] R. Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.
[54] R. Y. Rubinstein and D. P. Kroese. The cross-entropy method: a unified approach to combi- natorial optimization, Monte-Cate Carlo simulation, and machine learning, volume 133. Springer, 2004.
[55] S. G. Bardenhagen and E. M. Kober. The generalized interpolation material point method. Computer Modeling Engineering and Sciences, 5(6):477-446, 2004.
[56] Z. Huang, X. Lin, and D. Held. Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. 105:10..1607//SS.220222 202.XXSIS..22...11..1.1.
12

---

Supplementary Materials
6.1
Inputs and Outputs of RoboCook's Respective Modules
13
6.2
Perception
14
6.2.1
Data Collection
14
6.2.2
Data Preprocessing
14
6.3
Dynamics Model
14
6.3.1
Graph Building
14
6.3.2
Model Training
14
6.3.3
Building Synthetic Datasets
14
6.4
Closed-loop Control
15
6,4.1
Action Space
15
6.4.2
Multi-bin Classification
16
6.4.3
Subgoal Definitions...
16
6.5
Experiment Setup
16
6.5.1
Robot and Sensors
16
6.5.2
Tool Design
16
6.5.3
Tool-Switching Setup
17
6.6
Comparison with Human Subjects on Humpling-making
17
6.7
Tool Classification
18
6.8
Human Evaluation of Alphabet Letters
19
6.9
Baseline Implementation Details
19
6.10
Analysis of Comparison with Baselines
19
6.1 Inputs and Outputs of RoboCook's Respective Modules
Perception module: At each time step. the input is observation point cloud merged from four RGBD cameras and the franka panda end-effector pose. The output is clean and sparse surface point cloud representing the dough concatenated with point cloud sampled on the tool mesh surface representing the action. We compute the pose of the tool point cloud from the end effector pose and the tool geometry.
GNN dynamies: Our GNN predicts S.41 from S, and a,41 at each prediction step. The inference- time input is the initial state So and {ac, 1,2,.... N). The output is Sy. We select N 15 for the gripping, pressing, and rolling tasks.
Tool predictor: The input is point cloud of the current dough configuration point cloud of the final target (c.g., the dumpling point cloud). The output the next tool achieve the final target configuration.
The input is point cloud of the current dough configuration and the point cloud of the subgoal associated with the selected tool. We have a one-to-one mapping between subgoals and tools. For example, a rolling pin corresponds to a dough flattened by the rolling pin. The output action in the defined action space of the selected tool.
Policy network:
13

---

6.2 Perception
6.2.1 Data Collection
To train the GNNs, we collect around 20 minutes of data for each tool using a behavior policy that randomly samples parameters within a predefined action space. In each episode, the robot applies multiple action sequences to the soft object, and after the current episode, a human manually resets the environment. Humans reshape the dough into a bulky but not necessarily cubic shape after every five grips, three rolls, and three presses.
The data collected for each tool are as follows: (1) Asymmetric gripper two-rod symmetric gripper two-plane symmetric gripper: 60 episodes with five sequences per episode; (2) Circle press square press circle punch square punch: 90 episodes with three sequences per episode; (3) Large roller / small roller: 80 episodes with three sequences per episode.
We collect point clouds before and after executing each sequence to train the tool classifier. How- ever, we augment the training data by including any pair of these point clouds, not just consecutive pairs. For tools that don't 't require a GNN-based dynamics model, we execute a pre-coded dumpling- making pipeline ten times and add point clouds captured before and after using each tool in the pipeline to our dataset. Note that most tool selection data direct reuse of the dynamics data collected during the training of the dynamics model. This approach efficiently collects real-world tool selection data without needing extra exploration. We record the data collection process in the fourth supplementary video.
6.2.2 Data Preprocessing
When building the dataset to train the dynamics model, aside from the sampling process of the perception module at each time frame, we also want to leverage the continuity of the video data. Therefore, we introduce simple geometric heuristics into the physical environment for better frame consistency. First, if the operating tool not in contact with the convex hull the object point cloud, we use the same sampled particles from the previous frame. This also applies when the tool moves away from the object. Additionally, subsample the original videos to ensure that each video in the dataset has the same number of frames (16 frames in practice).
6.3 Dynamics Model
6.3.1 Graph Building
When building the graph, the edges between the dough particles are constructed by finding the nearest neighbors of each particle within a fixed radius (in practice, 0.1 cm). The edges between tool and dough particles are computed slightly differently. Instead of simply connecting to all the neighbors within the threshold, we limit the number of undirected edges between tool particles and the dough particles to at most four per tool particle neural network. Since all the node and edge features in the GNN encoded in each particle's local neighborhood, our GNN is naturally canslation-invarian and therefore can accurately predict the movement of the dough regardless of its absolute location in the world frame.
off the redundant edges in the graph
6.3.2 Model Training
We train the model with temporal abstraction to enhance performance and inference speed. For example, when = 0, we train the model to predict the state of the dough at f = 3 directly instead of = This shortens the horizon, eases the task, and improves our model's inference speed by decreasing the number of forward passes needed for a full action sequence.
6.3.3 Building Synthetic Datasets
Since the synthetic dataset generated by the dynamics model is cheap, can collect much synthetic data as desired. Empirically, for example, sample 128 random actions in the action
14

---

space of gripping for each of the 300 different initial states, so there are 128 * 300 = 38, 400 pairs of input and output point clouds to train the policy network. The random walk to generate the synthetic dataset starts from initial dough states acquired during dynamics data collection, which are not all block-shaped states and cover various shapes.
6.4
Closed-loop Control
6.4.1 Action Space
We classify the tools into a few categories based on |A], the dimension of their corresponding action space. We visualize action spaces for gripping. pressing, and rolling in Figure 3 (B).
A) Nine tools that have an action space with AA| 2 3:
1) Asymmetric gripper / two-rod symmetric gripper / two-plane symmetric gripper: {r, 0, d}, where r is the distance between the midpoint the line segment connecting the centers of mass of the gripper's 's two fingers and the center of the target object, the robot gripper's 's rotation about the (vertical) axis, and is the minimal distance between the gripper's
fingers during this pinch.
2) Large roller / small roller / square press square punch: {z, 0}, where {x, y, =} the bounded location indicating the center of the action, and @ is the robot gripper's rotation about the vertical axis. In the case of rollers, the rolling distance fixed and therefore not included in the action space.
3) Circle press / circle punch: {ar, y. =}. where Vz, y, =} is the bounded location indicat- ing the center of the action. The robot gripper's 's rotation is unnecessary because the tool's bottom surface a circle.
B) Five tools that have action space with |A| = 2:
Knife circle cutter pusher skin spatula/ filling spatula: {x, y), where {ar y) the bounded location indicating the center of the action on the plane. and are fixed for these tools to simplify the action space.
C) The action of the hook is precoded.
In category B, for all tools except the knife, we leverage the prior that center of the dough is always the optimal solution in the action space and directly compute the center from the processed point cloud. In the case of the knife, solution for (the zyz coordinate system is illustrated in Figure 4), For x, we first compute the volume of the target dough and then perform binary search with the center of the dough as the starting point to find the cut position that results in the closest volume to the target volume.
the y coordinate of the center of the dough as the
In category C, the hook precoded first to hook the handle of the dumpling mold, then close the mold, press the mold handle to turn the dough into dumpling shape, and finally open the mold by hooking and lifting the handle.
The guiding principle in designing action spaces involves starting with the end-effector's 6-DoF action space and eliminating redundant DoFs. For instance, rotations along the x and y axes are typically not required to generate meaningful action. Hence, we opt to exclude them from the action space of the 14 tools. For grippers, we transform the Cartesian coordinate system into polar coordinate system simplify the search process for action parameters since corner cases in the bounded Cartesian space are usually suboptimal. Following this, we introduce tool-specific DoFs, which are determined by the tool's 's geometric properties. For example, in the case of grippers, we incorporate an additional parameter, d, to represent the width between the gripper's two fingers.
Our method can potentially generalize to various challenging dough manipulation tasks besides dumpling-making, such making alphabet letter cookies (as shown in the paper), pizza, and noo- dles. A successful transfer requires the ground truth meshes of new tools and data from interacting with them. We only need 20 minutes of real-world interaction data per tool, demonstrating the ease
15

---

of retraining for new tasks and tools. Although we incorporate human prior knowledge to simplify the action space for tools, does not constrain the generalization capability since we can easily spec- ify the action space for new tools. One limitation is that hand-designed action spaces may not be desirable in general manipulation settings. Our insight that for most tasks, especially tasks where the robot uses tool, there is much redundant and undesired space in the full 6-DoF action space. In most cases, humans follow certain rules and constrained action space when using specifie tool. A future direction is to design a planner to automatically prune the action space conditioned on the tool and the task.
6.4.2 Multi-bin Classification
We formulate the self-supervised policy training as a multi-bin classification problem inspired by previous works on 3D bounding box estimation [51, 52]. The total loss for the multi-bin classifica- tion
|A| Σ
C
(
(5)
where the confidence loss Lannt is the softmax loss of the confidences of each bin for each action parameter A,, and the localization loss Lide the loss that tries to minimize the difference between the estimated parameter and the ground truth parameter. For orientation estimation, we use negative cosine loss as the localization loss and force it to minimize the difference between the ground truth and all the bins that cover that value. We use the smooth LI loss as the localization loss for action parameters not representing an orientation. During inference time, for each parameter, the bin with maximum confidence is selected, and the final output is computed by adding the estimated delta of that bin the center of the same bin.
To establish the bins, we first bound our action space into a reasonable range (Amin, Amaz) ). Second, we define the number of bins N as a hyperparameter and select N = for translations and N = 32 for rotations. Third, we divide the action space into N + bins with size * (Amaz Aman) \/N. The center of the first bin is Amin: the center of the last bin is at Amars and each bin overlaps with neighboring bins. This approach is similar [51].
6.4.3 Subgoal Definitions
As mentioned in 6.2.1, during data collection, execute hand-coded dumpling making pipeline ten times and add point clouds captured before and after using each tool in the pipeline to our dataset as expert demonstrations. The point clouds recorded after using each tool in one of these trajectories are selected as subgoals.
Experiment Sctup
6.5.1 Robot and Sensors
We use the 7-DoF Franka Emika Panda robot arm and its parallel jaw gripper as the base robot. Four calibrated Intel RealSense D415 RGB-D cameras are fixed vertical metal bars around the robot tabletop, as shown in Figure 4. The cameras capture 1280x 720 RGB-D images at 30 Hz. We also design set of 3D-printed tools based on real-world dough manipulation tools.
6.5.2 Tool Design
We design and 3D-print 14 tools: large roller, small roller, circle press, circle punch, square press, square punch, knife pusher, circle cutter, two-rod symmetric gripper, asymmetric gripper, two- plane symmetric gripper, skin spatula, filling spatula, and hook. The dumpling mold the same as real-world ones. In Figure 7, we compare our 3D-printed tools and their real-world prototypes, which are common kitchen tools for dough manipulation. The design principle of these 3D-printed tools is to mimic real-world ones as closely possible.
16

---

Prototype
3D-printed tools
Grippers
Rollers
Presses/punches
Knife
Circle cutter
Spatulas
Hook
Figure 7: Prototypes of 3D-printed tools. We show a comparison between our 3D-printed tools and their real-world prototypes which are common kitchen tools for dough manipulation. The design principle of these 3D-printed tools is to mimic real-world ones as closely as possible. We use 3D- printed tools instead of real-world ones to allow the robot arm acquire and manipulate the tools more easily.
The roller composed of a holder and rolling pin so that the rolling pin can rotate freely while the holder remains static. We designed both large and small rollers to accommodate different needs. We also have set of punches and presses with square and circle shapes. The knife is thin planar tool that can cut through objects. Similarly, the circle cutter can cut an object into a circular shape. Among the grippers, the two-rod symmetric gripper consists of two cylindrical extrusions, the asym- metric gripper consists a cylindrical and planar part, and the two-plane symmetric gripper consists of two planar parts. The two extruding rods on each gripper insert into the corresponding holes of the two fingers of Franka's gripper, allowing them adhere and move along with the fingers. linear shaft connects the two parts of each gripper, constraining their movement The skin and filling spatulas have similar design, except that their two extrusions are each spatula, so they can pick up and put down the soft object without deforming it. The hook and the dumpling mold are tools used together to mold the dough into dumpling shape.
single axis.
6.5.3 Tool-Switching Setup
The tool-switching setup is an engineering innovation we implement in this project. We adopt two designs so that the robot can pick up, use, and put down the tools without any help from humans: (1) The connector on the top of each tool attaches firmly to the Franka's 's gripper when it closes its fingers and also unlocks easily when the gripper reopens. (2) The tool racks on the bottom and right side of the robot table hold the 3D-printed tools in their upright poses so that robot can easily pick up and put down the tools. Additionally, we calibrate the tools' world coordinates so that the robot knows where find each tool. The supplementary videos of making dumplings show examples of how the robot switches tools.
6.6 Comparison with Human Subjects on cammpling-making
We invited five human subjects to make dumplings with the same tools highlight the complexity of dumpling-making. Fach subject participated in two experiments: choosing tools independently and following given tool sequence and subgoals. For a fair comparison, human subjects were not allowed to directly touch the dough with their hands or apply each tool more than five times. Before the experiments, we introduced each tool and gave them sufficient time to get familiar with the dough's 'ss dynamics and devise their plan. We compare their best attempt among three trials to our method for each experiment. Figure shows that human performance is notably worse than our method without subgoals. Performance improves with the tool sequence and subgoals but remains comparable to or worse than our method. The fifth supplementary video records the entire process.
17

---

Circle cutter
18
Knife
100
Asymmetric gripper
Two-plane gripper
118
Two-rad gripper
Hook
Circle press
Label
Square press
60
Circle punch
Square punch
-40
Pusher
Large roller
Small roller
B
0
20
Filling spatula
Skin spatula
Circle cutter
Knife
Asymmetric gripper
Two Two-plane plane gripee
gripper
Two rod gripper
Hook
Circle press
Square press
Circle punch
Square couarepuucch punch
Pusher
Large roller
Small
Filling spatula
Skin spatula
Prediction
Figure 8: Confusion matrix of the tool classifier predictions. We show the confusion matrix of the tool classifier predictions on the test set, which is split from the training data. The tool classifier achieves accuracy very close to 1.
Subject
Subject 2
Subject
Subject 4
Subject
RoboCook
W/o subgoals
W/ subgoals
Figure 9: Comparison with human subjects. We show a comparison with the manipulation results of human subjects. In the first row, Human subjects devise their manipulation plan and choose tools independently. In the second row, human subjects follow a given tool sequence and subgoals.
6.7 Tool Classification
The training of our tool classifier is supervised learning with cross-entropy loss in standard classification architectures. We split test set from the training data of the tool classifier and show
18

---

the confusion matrix of the tool classifier predictions in Figure 8. The instance accuracy is 0.996. We compared PointNet-based and ResNet-based architectures for the tool classification network. PointNet-base architecture generalizes better due to its ability to encode depth information. Empiri- cally, it demonstrates greater robustness to changes lighting, dough color, and dough transforma- tions.
6.8 Human Evaluation of Alphabet Letters
We recognize a discrepancy between how metrics such as Chamfer Distance measure the results and how humans perceive them these metrics are prone to local noises while humans are good at capturing the holistic features of the dough. Therefore, we invite 100 human subjects to evaluate the results. The human survey asks the question: "What alphabet letter is the robot trying to shape in the given image?" If we put all 20 images (four methods x five letters) in Question 1, there could be a predictive bias from seeing more than one image of the same letter. Therefore, we shuffle the order of 20 images and split them into four groups. Each group contains one image for each letter but from different methods. After averaging over five letters, show the human perceived accuracy and human ranking of the performance of these four methods in Table 1.
6.9 Baseline Implementation Details
Baselines RoboCraft, CEM+GNN, and RL+GNN use the same framework as RoboCook but replace the policy network with gradient descent (RoboCraft), CEM, and RL. In other words, GNNs in these baselines are the same as GNNs we trained in RoboCook. They also use the same tool predictor as in RoboCook to handle this multi-tool task. They are meant to compare the policy network in RoboCook with alternatives.
The RL+GNN baseline utilizes a model-based Soft Actor-Critic (SAC) with a learned GNN-based dynamics model as the world model. The action space aligns with other planning methods, and the state space comprises the point cloud position. The reward function is derived from the change in Chamfer Distance after each grip. Training involves discount factor of 0.99, learning rate of 0.0003 with the Adam optimizer, 2-layer MLPs with 256 hidden units, and ReLU activation for both policy and critic models. We initially collect 250 steps of warm-up data. The replay buffer size is 1e6, and the target smooth coefficient is 0.005. We trained the RL baseline for 2500 steps online for each action prediction to fit into our closed-loop control module. The inputs and outputs of the RL baseline are the same as our policy network. The input is a point cloud of the current dough configuration and the point cloud of the subgoal. The output the action in the defined action space of the selected tool. The results shown in Figure 5 and Table I indicate that the RL baseline is noticeably worse than method.
6.10 Analysis of Comparison with Baselines
Our methods outperform baselines RoboCraft, CEM+GNN, and RL+GNN by a large margin, using the same GNNs and tool predictors but different planning methods. One reason is that our policy network sees a complete coverage of the entire action space from the large synthetic datasets gen- erated by our dynamics model offline. Other planning methods explore the action space online and therefore have smaller coverage than ours. By training a goal-conditioned policy network offline, also make the online action synthesis much more efficient than baseline methods.
Another insight is that our PointNet-bused policy network is better at capturing higher-level shape changes of the point cloud, such as concavities. For example, by comparing the current and target dough configuration, the policy network knows that the gripper should probably pinch the concave locations in the target dough configuration to deform the current dough into the target shape.
Our method also outperforms the bascline CEM+MPM since MPM simulator suffers from sim2real gap and relies on careful system identification to bridge the gap. Thus, the MPM simulator underperforms the GNN, which is directly trained on real-world data.
19


Reviews:
[{"review_id": "m63F59IjmV", "reviewer": "Reviewer_WdqP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work develops an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. It demonstrates that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies.", "summary_of_recommendation": "This is a solid work with multiple merits. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- This work tackles important, novel and challenging manipulation tasks.\n\n- The paper is well-written and easy to follow. The figures did a good job illustrating the ideas.\n\n- The system is reasonably designed with insightful discussion on the various choices.\n\n- The experiments have been conducted extensively on real world hardware. The results are convincing.\n\n- The additional results on generalizing to different materials without retraining is quite interesting.\n\n\nWeakness and questions:\n- If I were to be nitpicky, there is a lack of baseline for comparison in the task dumpling making. However, it is understandable that finding a suitable baseline on these complex tasks may not be easy.\n\n- Another prior art on tool usage for long-horizon sequential manipulation [A] might worth discussion in the related work.\n\nOverall, this is a solid work with multiple merits. \n\nReferences:\n    A) Liang, J., Wen, B., Bekris, K. and Boularias, A., 2022, May. Learning sensorimotor primitives of sequential manipulation tasks from visual demonstrations. In 2022 International Conference on Robotics and Automation (ICRA) (pp. 8591-8597). IEEE.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "hmoNYgiwR3m", "reviewer": "Reviewer_VnEP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes RoboCook, a framework for manipulating deformable material (e.g., dough) with a set of available tools. They propose a novel tool selection approach by self-supervising a classifier on random tool interaction results. Given a subgoal pointcloud, the classifier selects the tool and either a goal-conditioned tool-conditioned action policy or a hand-designed policy is executed to reach the subgoal. Policies are trained using random tool action rollouts. The experiments show that the method can be used (given subgoal images) to create a dumpling and shape dough into arbitrary shapes (e.g., letters) and outperforms several baselines on the latter task.", "summary_of_recommendation": "While I am not sure that their claim to \"long-horizon planning\" is justified, due to the prescribed subgoals, their full system, self-supervised to simultaneously train dynamics, policies, and the tool selection policy seems novel and effective for a difficult set of tasks and is nicely demonstrated with high performance compared to existing baselines. The paper needs some clarity on method details, but I believe presents a novel and significant result.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What are the precise inputs/outputs for each component of RoboCook (e.g., GNN dynamics, tool predictor, and policy network)?\n- On what data are the baseline models trained, in particular baselines 1 and 2? How long is the RL baseline trained and what are the precise inputs/outputs?\n- How are subgoals collected here? Video seems to suggest that the method may be able to realize when it needs to backtrack to an earlier subgoal, but does not specify how this is/could be done.\n- Is tool identity input to the network or is there an individual policy per tool?\n- How are the bins for the action policy established?\n- In what frame are particle locations localized in as inputs to the GNN? If global frame, how is translation invariance achieved?\n- In Sec. 6.1.2, what is mean by \"optimizing the sample quality at each time frame\"?\n- In Sec. 6.2.2, the \"temporal abstraction\" is unclear it unclear how this is related to lines 111-112 in terms of prediction horizon. Please provide clarification.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Excellent system demonstration; able to perform two difficult tasks (making dumpling and letter shaping).\n- Method is well formulated, runs fast, and seemingly trained fully from randomized data.\n\nWeaknesses:\n- Relies upon provided subgoals to achieve long horizon planning.\n- Human priors on actions space are difficult to scale to new tools/tasks.\n- Paper is lacking important details w.r.t. the proposed method and the baseline.\n- Fixed set of tools.\n\nThis work proposes a novel framework that builds on existing work modeling deformable object interactions with Graph Neural Networks (GNNs) (e.g., [18]) by adding a tool selection model that forwards to either a GNN based low-level policy or a hand-coded policy. This enables their approach to effectively perform many subtasks required to solve complex tasks such as making a dumpling.\n\nThe authors claim that RoboCook enables long-horizon tasks, however, their method seems to be highly reliant upon subgoal images that guide the robot though the task. As such, individual policies really perform short-term goals and no long-horizon reasoning seems to be taken place. The authors should clarify how RoboCook enables long-horizon reasoning.\n\nThe description of the proposed method is somewhat difficult to follow and is missing important details. The authors should be clearer in the introduction that their framework also uses hand-coded policies for certain tools (though the tool/policy is still autonomously selected). Further details on each component should be given, specifically, what exactly are the form of the inputs/outputs of each model component. For instance, are edge features considered in the GNN? Is the tool identity input to one network or is there a network for each tool? As a point of reference, [18] eqs. 2-5 help clarify the inputs/outputs of their GNN model; a similar style of network description could really help clarify the methodology here.\n\nFurther clarifications on the baseline methods are needed to clarify the significance of the results. In particular, more details on what data baselines 1 and 2 are implemented with (still multi-tool?) should be provided. The inputs/outputs of the RL baseline is also difficult to understand, even with the appendix description.\n\nA limitation of this work is also the hand designed action spaces. Here, the class of task makes it so that actions can be easily specified, but generally speaking it can be difficult/undesirable to specify tool action primitives (e.g., consider multiple-objects in scene or full 6-dof control). This somewhat limits the impact of this work.\n\nThe proposed method is able to select between tools/policies effectively and all from self-supervised random interactions. Being able to learn autonomously from simple/small self-supervised data collection policies is a significant result for tool use. However, it does here assume guided subgoals to complete the longer task. Discussing how future work might loosen this dependency further could help solidify the significance of the work. Similarly, how could future work enable method to generalize to novel tools during runtime?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "DkCbwDLGzTk", "reviewer": "Reviewer_bNTe", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors propose a real-world robotic system and learning framework for manipulating elasto-plastic objects (namely, dough) over long horizons and using tools. The paper introduces three components which they use as building blocks for the long-horizon policy. First, they propose to learn a forward dynamics model of dough/tool interaction from raw point-cloud input using a GNN. Second, they learn a model to select the appropriate tool for the current stage of the manipulation. Finally, they leverage the dynamics model to train a policy (may be viewed as an inverse dynamics model) which generates the action to move between two different dough states using a self-supervised learning procedure. The authors then design two long-horizon dough manipulation tasks (dumpling making and alphabet shaping), and convincingly demonstrate that their system is able to accomplish these difficult tasks in a real robot environment.\n", "summary_of_recommendation": "I really like this paper - it's a really strong example of a well-designed system which completes a very difficult task and incorporates learning methods in reasonable ways. The attention to detail in the system design, in particular, is something that many roboticists should aspire to. It's also the first time I've seen a full dumpling-making setup w/ learning which wasn't entirely scripted. \n\nWhile the subcomponents don't have much technical novelty, their integration in a larger system which actually works warrants acceptance of this work to CoRL, if only to serve as inspiration for researchers in the robot learning community to consider any of the (many) intermediate steps of dumpling-making, which each highlight unique challenges which independently warrant investigation. Being able to see system-wide limitations is excellent (and doesn't happen nearly enough, as much research tends to be quite narrow, particularly in deformable manipulation).", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Not so many questions, mostly just a request for more clarity on the following from above:\n\n- The authors don’t really discuss how the goal states are generated for each substep of the manipulation pipeline.\n- Using s=2 as a trajectory length doesn’t really feel like a long-horizon prediction: perhaps motivate this better in the main text?\n- How is the environment reset during random exploration?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Originality \n\n## Problem Statement\nThe problem of robot learning for dough manipulation is relatively new, but with a reasonable amount of investigation in the last few years. It is an interesting and impactful area of research - however the authors are not the first to propose dumpling making or letter-shaping as tasks. Additionally, most/all of the individual subtasks proposed for dough manipulation (cutting, pushing, rolling) have been incorporated into robot systems before ([1], for instance).\n\n[1] Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation, cited in the work. \n\n## Methods: \n\nThe three components of the method are relatively well-understood techniques in the community:\nLearning dynamics models of particles in deformables or liquids, using GNNs, is a common approach. However, the authors propose a very nice methodology for extracting clean+consistent point clouds suitable for training a dynamics model from real-world data.\nTool classification: the authors use a standard classification architecture to choose a tool - whether the training of this classifier used anything non-standard (self-supervised, or novel loss/dataset) was a bit unclear from the main paper.\nSelf-supervised, goal-conditioned action prediction: learning an inverse dynamics model to use as a goal-conditioned policy is a well-studied technique, as is learning parameterized skills for each tool. I haven’t specifically seen them used together, but it’s a natural design in a long-horizon, multi-skill setting where intermediate steps are well-understood.\n\nI would classify this paper as a novel (and high-quality) arrangement of many standard learning+robotics techniques that enables very impressive dough manipulation tasks.\n\n# Quality:\nThe quality of this submission is extremely high. The robotic system used is quite impressive: \n1. the use of multiple cameras to extract a clean point cloud of the dough which is good-enough-quality for training a dynamics model is no small feat, even if some of the perception system is made easier using color segmentation.\n2. The tool design and integration is quite extensive and well-thought-out, as are the interfaces between the robot+tool and tool+action space. And the tool swapping looked quite smooth. Overall an extremely impressive robotic setup.\n3. Each submodule seems to have been implemented with care and thorough/thoughtful design.\n\nSome small design issues I noticed:\n1. Using the Earth-Mover’s distance as a part of the loss when you have occlusions between start/end frames seems like a recipe for difficult convergence / poorly-defined loss metric, since the start points are (presumably) evolved to their final locations via the GNN, even though the initial points may no longer be visible in the final observation.\n2. The perception system relies quite heavily on knowledge of the environment, including color + geometry + full knowledge of tools.\n\nAlso, the videos are great.\n\n# Clarity:\nThe submission is quite well-written, and well organized. There were a couple of unclear parts:\n1. The authors don’t really discuss how the goal states are generated for each substep of the manipulation pipeline.\n2. Using s=2 as a trajectory length doesn’t really feel like a long-horizon prediction: perhaps motivate this better in the main text?\n3. How is the environment reset during random exploration?\n\n# Significance:\nI’d say this work is reasonably significant, because it provides a pretty compelling system which has impressive performance on a notoriously difficult manipulation task. While I don’t think that any of the learning components used in the system are substantially different from prior work (and many of the subcomponents have been used even in prior dough manipulation work), it plants an impressive flag - I don’t think I’ve seen a system which completely makes dumplings before with as few constraints on the dough (size, initial position, etc.) as in this work.\n\nI think the design of the tools, their action spaces, and describing a full start-to-finish dumpling making procedure which uses them is in-and-of-itself valuable, because it provides an excellent starting point and point of reference for other researchers looking to improve a full working dumpling-making system. While it falls short of providing a benchmark set of subtasks (which I think would be a very cool contribution…), it would likely be easy for others to compare submodules with the performance of this system (and also ground the utility of others’ submodules).\n\n# Relevance:\nDefinitely a relevant paper, because it contains both a real robot and real-world learning, and on a very challenging task. The CoRL audience will definitely be interested in the paper, although perhaps more for the system design than for any of the algorithms used/proposed.\n\n# Limitations:\nThe paper lists a few limitations, which all boil down to the two following categories:\n1. The policy is following a high-level script, and therefore cannot recover from failures (either system-wide or even back-tracking). For specific, noteworthy tasks like making this type of dumpling that might be fine, but modifying this script for other variations (different size dumplings, different crimping, different filling, etc.) would be highly manual.\n2. Limitation to dough - other deformables with other constraints would require a very different perception + dynamics-modeling system.\n\nThere are two other substantial limitations not mentioned, in my opinion:\n1. The perception system makes many simplifying assumptions which reduce it’s generality - knowing dough color, table shape, exact tool sizes, etc. Allow the authors to dodge difficult perception problems - and it’s not exactly obvious to me that the system would scale to more complex dough scenes (i.e. where there’s a lot of dough being worked with, cut up, etc.).\n2. There’s a rather inherent limit on the expressivity of the sub-policies. Using primitives controlled at low-frequency with no real tactile/proprio or otherwise closed-loop feedback during the execution of the trajectory limits the complexity of tasks this system could be incorporated in. The only piece of the system which would really require this kind of feedback when executed by a human (crimping) is offloaded to a specialized tool (the metal crimping device). \n\n# Strengths/Weaknesses\nStrengths:\n- Excellent technical system, thoughtfully executed\n- Famously difficult domain with a challenging long-horizon task\n- The system works!\n\nWeaknesses:\n- Not much novelty of sub-components\n- The method evades some of the more-interesting problems in dough manipulation (i.e how to control dough w/ feedback at very-high frequency, maybe with additional sensory modalities) through careful design of tools and processes.\n- Requires a human-defined script and subgoals.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "AMB0sJ6DGAP", "reviewer": "Reviewer_UkA9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a system that can choose appropriate tools to shape dough in making dumplings and letter-shaped cookies. The system builds upon a GNN learned from interaction data for each tools that model the dynamics, and a mesh registration and re-sampling process to map the visual observation to the particle representations. Instead of planning through the learned dynamics, the authors propose to learn the inverse dynamics given the goal shape by synthesising from the learned forward dynamics, thus significantly reduce the planning time for a closed-loop solution. An additional model is trained to infer the most probable tool to use facing the current observation and target shape. The results show that the learned dynamics is generalisable from dumpling making task to the cookies task and cookies shape with different materials.", "summary_of_recommendation": "The work presents an interesting system dealing with a task with considerable complexity. The proposed methods are not ground-bleakly novel while are reasonably composed and sufficiently tested in empirical studies. Some more in-depth discussion and analysis could make the framework stronger. However, its current form is interesting and thorough enough as a system paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Figure 3: since the synthetic data-set is generated from randomly sampled action, it is unclear how large the dataset should be and how long the exploration should be to sufficiently cover states that are similar to target shapes. Does this random walk always start from the block-shaped state? Will this become a scalability issue if more tools or finer target shapes were considered?\n\n* Line 133, \"closed-loop control with visual feedback\": it is not clear which control rate is the control operated on. Is it per step or per tool stage? Is it sufficiently efficient to run the mesh reconstruction and re-sampling in real-time to support feedback at this level? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n* Challenging long-horizon and deformable object manipulation task;\n* Extensive studies with comparison to baselines and a generalisation task.\n\nWeakness:\n* Limited method novelty;\n* Unclear that how large the datasets are for a sufficient coverage of state space;\n* Lacking insights to interpret the performance difference in Table 1.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a method for manipulating deformable objects with a set of tools. The authors propose a classification-based tool selection method, which, given a target point cloud, selects the tool and a policy. The experiments demonstrate that the method outperforms baselines for dough-shaping tasks.

Strengths:
- A strong set of experiments shows the method works for two difficult tasks
- Especially impressive is the extensive use of real-world hardware
- It is impressive that the method works while only being trained on random data
- The paper is well-written.

Weaknesses:
- Subgoals are necessary to achieve non-trivial tasks
- Some domain knowledge regarding action spaces is needed
- Unclear if this method will generalize to new tools

Please see the reviewers' comments for more information.
Decision: Accept (Oral)