ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
Authors: Daniel Seita
Yufei Wang
Sarthak J Shetty
Edward Yao Li
Zackory Erickson
David Held
Venue: CoRL 2022 Poster (2022)
Keywords: Flow
Point Clouds
Tool Manipulation
Deformables

Abstract: Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information.
In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. 
We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform physical scooping experiments with ToolFlowNet and find that we can attain 82% scooping success.
See https://sites.google.com/view/point-cloud-policy/home for supplementary material.

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
Daniel Seita, Yufei "aangg, J. SSettyy, Edward Yao Lit, Zackory Erickson, David Held 'Equal contribution. The Robotics Institute, Carnegie Mellon University, USA
Correspondence to: dseitaCandrew. wwtrrewww..eee.
Abstract: Point clouds are widely available and canonical data modality which convey the 3D geometry of scene. Despite significant progress in classifica- tion and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works imitation learning focus on learn- ing policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use novel neural network. ToolFlowNet, which predicts dense per- point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imita- tion learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 phys- ical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://ttnyuul.coo cwm/toolflomoe for supplementary material.
Keywords: Flow, Point Clouds, Tool Manipulation, Deformables
Introduction
Recently, learning-based techniques have become effective for improving the generalization capa- bilities of robots for manipulation tasks such as grasping [1], reorienting [2], rearrangement [3], and tossing [4]. Data observations tend to either images [5, 6, 7] or state information such as joint angles or end-effector poses [8], which are passed into a deep network obtain an output vector encoding action, typically representing change in end-effector pose joint angles. While these approaches have shown a wide range of successes, a fundamental limitation has to do with the nature of the observation. Using images requires projecting information into a 2D space which might lose valuable 3D information. Furthermore, learning from images in simulation often leads to a sim2real gap [9] in performance. Although is easy to access the internal robot states such as joint angles, the robot does not necessarily have the ground-truth state of objects in the environment, which might require complex state estimation systems [10]. Moreover, it hard define state for deformable objects like liquid and cloth [11, 12].
In this work, we propose framework for learning robotic manipulation from point cloud obser- vations. Point clouds are canonical data modality and are widely available from camera sensors, providing valuable information about the structure the 3D space [13, 14]. However, policy learn- ing from point clouds has been less explored compared to learning from images or state, potentially owing the difficulty of reasoning about raw 3D point cloud inputs. While there have been many proposed architectures which are specialized for learning from point clouds [13, 15, 14, 16, 17], these works tend to focus on computer vision tasks such as classification and segmentation. Policy learning from point clouds, while feasible in some contexts [18, 19], remains challenging.
We study learning from point clouds for robotic manipulation tasks with tools. The input data segmented point cloud which, for each point, contains its 3D coordinates and one-hot vector
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Before
Tool Flow
After
Before
Tool Flow
After
Not applied on a pouring task in simulation, where the tool the which contains water, Given a point cloud (colored blue), ToolFlowNet learns dense per-point flow vectors (colored red), which describe the intended 3D motion of each tool point. These are converted to translation and rotation actions. Left: the tool moves upwards. Right: the tool rotates pour water. We subsample the flow for visual clarity.
Figure I:
indicating the object class the point belongs Our key insight is use dense representations and flow to represent the tool action. We build upon dense point-cloud processing architectures [15] and train the model to predict per-point output values which we call tool flow. This represents the 3D movement of each tool point in point cloud from one time step to the next, which is an instance of scene flow [20]. Our model is trained with Behavioral Cloning on tool flow data, which provides dense per-point supervision. Given the set of tool flow vectors, we convert flow to an SE(3) transformation, which represents the actual action robot would execute. We call this model ToolFlowNet and visualize it in Figure for pouring task in simulation. We compare this against non-dense methods which directly regress to action and demonstrate the benefits of tool flow as an action representation. To summarize:
We propose a general framework for learning from segmented point clouds for manipulation with tools by utilizing novel architecture, ToolFlowNet, which predicts per-point tool flow vectors. We show how train ToolFlowNet for imitation learning and explore different loss functions for training. We perform extensive ablation studies to validate these choices.
We perform simulated imitation learning experiments on scooping and pouring tasks and show the benefits of using ToolFloWNet over baselines which do not use flow.
We demonstrate ToolFlowNet achieves 82% success rate on 50 physical scooping trials.
2 Related Work
Point Clouds and Flow Researchers have proposed a variety of architectures specialized for learn- ing from point clouds [13, 15, 14, 21, 17, 22, 23). We aim to explore policy learning for robotic manipulation from point clouds, and the approach we propose compatible with any architecture producing per-point outputs from point clouds. Optical flow [24, 25] and its 3D counterpart, scene flow [20, 26], are widely used in computer vision, particularly in autonomous driving setups where the objective is to associate the movement of each pixel (or point in 3D space) from one image (or point cloud) to the next time step. We use flow as an action representation for robot manipulation, and our method could integrate prior flow estimation techniques if necessary.
Policy Learning from Point Clouds or Flow for Robotic Manipulation Learning from point clouds has been explored in grasping [19, 27], in-hand manipulation (by voxelizing) [18], visual navigation [28], and shaping 3D deformables [29]. Our work differs that we study tasks that involve manipulating tool in 3D space from point cloud observations, and where we use tool flow as the action representation to improve learning. While Qin et al. [30] extract tool point clouds and learn keypoints for grasping and manipulating tools, instead predict dense tool flow for manipulating the tool. Some prior work has explored policy learning using flow for robot manipu- lation, such as for fabric folding [31], manipulating articulated objects [32, 33], and manipulating 3D deformables [34]. This work differs in that we propose a more general framework that does not assume a specific structure the objects being manipulated, and which predicts flow on the tool a robot controls instead of flow on a target object. Furthermore, unlike prior work [35] which iteratively minimizes flow with pick and place actions, or other work [36] which uses optical flow flow to derive continuous tool motions in 3D space from visual input. A recent work [37] estimates optical flow using RGBD images from the current frame the demon-
on tactile sensors,

---

Position
Point Class
Tool Point
Flow
Inpot Feutures
Flow
Tool Flow
F tool
Manipulation
Tool Transformation
Segmentation
*ointNet**
Differentiable Layer
(R,
Backbone
Segmented
Point Cloud P
The proposed ToolFlowNet framework learns from segmented point clouds, which form the input to dense point cloud network produce per-point flow vectors. We extract just the tool points (holded above for clarity) use those tool points to determine transformation that robot should apply to the tool. See Section for further details. Above, we show the physical scooping task; see Section 4.4 for details.
Figure 2:
stration and extracts transformation to align them. In contrast, do not use flow for aligning frames to demonstrations but for deriving the transformations the tool should follow.
Deformable Object Manipulation We apply our proposed tool flow framework on tasks with continuous control of tool for deformable object manipulation. Such manipulation is challeng- ing for robots for reasons such as the difficulty in specifying concise state representation for de- formables and their complex dynamics [11. 12]. We test on scooping and pouring. Variants of these tasks have been studied in prior work. For example, [38] scooping
example application for task and motion planning, and [39] test scooping of granular media using 2D image representation. Unlike these works, our approach is general framework for robots performing continuous control of tool to manipulate deformables 3D space. Prior works [40, 41, 42, 43, 44] propose methods to detect or model physics properties granular media or liquids and test on scooping and pouring. In contrast, we propose general method of predicting 3D tool flow which does not require modeling properties of deformables and is not specialized scooping or pouring tasks, and which uses point clouds as inputs instead of RGB images [45].
3 Method: ToolFlowNet for Behavioral Cloning from Point Clouds
We consider policy learning from segmented point cloud observations. A segmented point cloud Pe at time / is N x d array with N points, each with feature dimension d. The feature of the ith point p E P. consists of its 3D coordinate position and a one-hot vector indicating the class of the object to which p(o belongs. For ease of notation, we suppress the time subscript and the point index superscript when the distinction is not needed. study Behavioral Cloning (BC) [46] from segmented point clouds. BC assumes access pairs (o,, a;) from a demonstrator, where o, indicates any type of observation (of which segmented point clouds are example). BC performs supervised learning to learn a policy и with parameters 0 such that the predicted action å, = (o:))) is close to the ground truth label a;- While prone to compounding eтгоrs at test time [47], BC has shown surprising effectiveness when compared to more complex learning- based algorithms in some robotic manipulation contexts [48, 49], which motivates further study on how it can be done with point cloud observations. In future work, we will explore combining our method with other imitation learning algorithms [50, 51, 52].
dataset D {(o,, ai) 3 of observation-action
3.1 Tool Flow As Action Representation
We propose to use tool flow as an internal representation for the action, where the flow ((i) R3 associated with point p(.) is 3D vector. For given tool point, we interpret its flow vector as representing how the point will move in 3D space as a result of "applying" the flow. To form the policy πy. we use a dense point cloud network (such as a segmentation PointNet++ [15]), which given an input point cloud produces per-point outputs. The point cloud input is already segmented in that it contains, for each point, the 3D world position and one-hot encoding of its class. With an (N x d,) -sized point cloud P input, the output F has dimension (N d2), where dy is the output dimension (in case, d2 3). We then extract from the output F the subset of N' < N
3

---

Predicted Tool Flow F tool
D'D-Projcted Tool Flow
Ground- -ruth Tool Flow
Apply Tool Transformation (R,
Point Matching Loss
Differentiable SVD Layer
Consistency Loss
Figure 3: A visualization of the proposed point matching loss (Eq. 3) and consistency loss (Eq. 4). The black points visualize a simplified ladle's point cloud, and the thin red arrows represent the flows the tool points.
points in P corresponding all points on the tool, while ignoring points belonging to other object classes. This results in a set of predicted 3D tool flow vectors Frool { with f(i E IR's for each tool point.
Suppose that, at time i, the expert applies an action to the tool which is given by a ground-truth transformation a = (R', ; t²) = T E SE3). Let tool C P be the set of 3D points on the tool. Then the ground-truth tool flow is given by ggt = T TP"Ptool tnol tool where P toot is the result of applying the transformation T on all points in P tool] Thus, there is one-to-one correspondence between the transformation T and flow ge: nonetheless, we show in Section 4 that estimating the tool flow leads to improved performance compared to estimating the transformation T directly.
Given the set of predicted 3D tool flow vectors F toal, the next step is to extract a single overall action å where à is a transformation that represents the change in translation and rotation of the tool's pose. To compute the action, we consider the tool point clouds Ptool and tool = P tool + Foool. where in estimate the best-fit
the latter, move each point based its estimated flow. Our objective tool transformation T (R, t) which contains rotation and translation components, respectively, to align P cool and P Yoool i.e., we R. we first center the two tool point clouds to obtain Ptoul and P nnol We then input the centered point clouds to a differentiable, parameter-less Singular Value Decomposition (SVD) layer [53, 54] which computes the rotation which best aligns P Proo and P Ptoot with respect mean square er- ror (MSE). The change in translation é can then be computed as = C(P RC((P (too)), where C(P) denotes the centroid of the point cloud P. By combining the translation and rota- tion components, we produce the full transformation T, which we treat as our action representation for the policy. The outputs for the non-tool points are not supervised. We call the resulting point cloud-to-action system as ToolFlowNet (see Figure 2), which can be used by robot for manipula- tion. Mathematically, let Fg represent the segmentation PointNet++ that generates the flow vectors. FoolFlowNet computes the tool transformation as follows:
find T to minimize |TPtool too! P P'
2 To obtain the rotation
â [R. (o) SVD(Ftoo)) tool)
where SVD represents the parameter-less Singular Value Decomposition layer described above, and F Frool is the flow corresponding to the tool points in the estimated flow F = (P)..
3.2 Imitation Learning via Tool Point Matching and Consistency Losses
Point Matching Loss: Given the policy's predieted action å TT(O), straightforward way to imitate the ground truth action a* = (R*,t") is to use the MSE loss:
Linse (a, a²) = 31 RŘ R + 32 t" |2
(2)
where and 82 are weights for the translation and rotation components. Instead of trying balance the weights, in this paper, use point matching loss reduce the diserepancy between å = (R. i) and the ground truth action a" = (R", t²). Given the predicted action, the transformation (R. t) is applied on all the original N' tool points in the point cloud, and the loss function Lpoint computes the Euclidean distance between the tool points transformed using the predicted action

---

(R. t) versus the tool points transformed using the ground-truie truth action (R",t") t*):
1 N Σ (Rp 19 +8) -(R"p + t2))l2:
Leoiin P. a,,,a"))
(3)
where p iterates through the N'' tool points in Prool S P. and we interpret R and RR represent- ing 3 x 3 rotation matrices. Prior work on 6D pose estimation [55, 56, 57] has used variants of this loss function to jointly optimize for translation and rotation as compared to balancing the weights on separate translation and rotation losses. Our usage of Lpoint is similar to that in Wang et al. [19] where the matching loss on tool points directly controllable by the robot.
то learn SE(3) pose changes (and thus, actions), its effect on optimizing the predicted flow vectors F happens via backpropagating through a differentiable SVD layer which "compresses" all predicted flow vectors to produce a single trans- formation (R. t). This compression means that there could be significant noise in the individual flow vectors, even they combine to form a reasonable action. Thus, we propose a consistency loss Tecomsistency which serves as a regularizer to ensure that the predicted flow vectors are similar their induced, SVD-projected flow vectors produced from the transformation encoded in å. The loss is:
Consistency Loss: While Lpoint should allow the policy
N
Leonsistency (P, a)
N Σ
[ (1) + - - -f
(4)
where for each of the N' tool points, compute Rp + i-t-pp (0)) the induced flow from the predicted transformation (R, t) after applying the SVD layer, and f()) is the flow predicted by the network before the SVD layer. Note that the ground truth transformation (R* t²) does not appear in this consistency loss. The consistency loss is only a function of set of points and a set of corresponding flow vectors on those points, and does not rely on any other ground truth signal. We combine this with the point matching loss Lapin to obtain the final loss function to optimize the policy Te. Teambo = Tppmim + A Trunmsitencys with hyperparameter À controlling the weight of the consistency loss, which we set to from traditional optical flow and scene flow methods, ToolFlowNet uses flow representation to compute the transformation of the tool, and is trained using the ground-truth demonstration action. It is not used to just estimate the flow.
0.1. See Figure for visuals. To distinguish our method
Additional Implementation Details:
To obtain ground truth tool flow Fgt in simulation, we de- termine the 3D movement of each tool point as result of applying the demonstrator's action to transform those points. In physical settings, we scan the tool to obtain 3D model, from which extract tool point clouds P tool- We perform similar calculation where we detect the transformation executed by the robot and apply it to obtain the flow for each tool point. This method of extract- ing F gt only requires access to the current observed point cloud and the corresponding action. In particular, it does not require the perhaps more restrictive assumption of requiring one-to-one point correspondence between two consecutive point cloud observations. In addition, this method to de- tect flow means that it reflects the "intended" action from the robot, which may differ from the true positions of the tool points in 3D space after the robot executes the action; for example, when collision happens with wall, the tool points might not move, even though the robot intended for them to move. We leave alternative techniques to extract tool flow to future work.
4 Experiments
4.1 Simulation Experiments
We build on top of SoftGym [58], which provides suite of deformable manipluation tasks and uses NVIDIA FleX [59] as the underlying physics engine. We use the simulator to obtain ground-truth segmentation labels. For the tool, use the "observable" point cloud at each time step, so there may be occlusions. We test two tool-based simulation tasks, Pour Water and ScoopBall, and for each, test
two action spaces: 3D and 6D for Pater, and 4D and 6D for ScoopBall. In PourWater the agent
5

---

Method
Loss
Dense PN++?
N. Success
N. Success
N. Success
N. Success
Average
ScoopBall
ScoopBall
PourWater
PourWater
Success
PCL Direct Vector
MSE
X
0.544:1:0.03
0.848:10.05
0.530:1:.08
0.402.10.04
0.581
PCL Direct Vector
PM
X
0.228:10.11
0.048:10.04
0.132:10.07
0.088:1..04
0,124
PCL Dense Transfomation
MSB
0.519:+0.07
0.824:2.06
0.539=10005
0.344:0000
0,556
PCL Transfomation
0.367.10.07
0.360
0.583.10.03
0.049.1.022
0.340
D Direct Vector
MSE
K
090:
0.952:10.02
0.035:1:0.01
0.069=+0.02
0.311
D+S Direct Voctor
MSE
X
0.734=0011
0.928=00003
0.777=#0.03
0.304 03
0.686
RGB Direct Vector
MSE
0.354:+0.05 X
0.776=:0.05
0.696:100.22
0.324:20.05
0.538
RGB+S Direct Vector
MSE
X
0.671
0.944.1000
0.804
0.353
0.693
RGRD Direct Vector
MSF
0.418:10.110
0.920=00002
0.733:::007
0.353:10.02
0.606
RGBD+S Direct Voctor
MSE
0.734 10
0.968:00002
0.830=
0.401
0,753
ToolFlowNet, No Skip Conn.
PM-C
0.987:10008
0.304
0.000:10.00
0.000:10.00
0,323
ToolFlowNNel. MSE after SVD
MSE+C
0.089:10.04
0.792.10.009
0.494:10.02
0.913::0005
0.572
Tet,FlowNet, PM before SVD
PM
0.785:1:0.08
0.880:1:0005
0.618:10.04
0.677:100.05
0,740
NoolFlowNet, No Consistency
PM
0.861=:0.06
0,744=2011
0,468::0.10 10
0.609::00006
0.670
ToolFlow.Net (Ours)
PM-C
1.152:10.07
0.952:10.02
0.795:10.05
0.667-1-0.03
0.892
Table I: Behavioral Cloning (BC) results in simulation. The first 10 rows are baselines, the next 4 are ablations of our method, and the last row is our method. We report the loss functions used MSE only, PM only (the loss in Eq. 3), or if it also uses consisteney loss (+C, from Eq. 4). We also show whether the method uses a segmentation PointNet+++ (i.e.. a dense architecture), and the normalized success rates (N. Success) across all tasks over independent BC runs. The last column averages the success across the four columns. We bold the best numbers in the columns, plus any with overlapping standard errors.
controls a box which contains water and must pour the water into a fixed target box. In ScoopBall, the agent controls a ladle and needs to scoop ball. See Appendix A.1 for more details.
4.1.1 Baseline Methods
We compare the proposed method with the following baselines (see Section 4.3 for ablations):
PCL Direct Vector. Uses a classification PointNet++ network directly estimate vector action (with translation and an axis-angle rotation). We test variants, one which supervises with the MSE loss and another which uses the Point Matching (PM) loss from Eq. 3 on tool points. PCL Dense Transformation. Uses a segmentation PointNet+++, and directly predicts per-point 6D vectors (translation and axis-angle). Each point cloud has a designated point as the center of rotation for the tool, and we use the output corresponding to that point as the vector action. The outputs for the other points are not supervised. This baseline is designed to isolate any ben- efits from using the segmentation version of PointNet++ instead of classification. As with Direct Vector, we test two variants, with supervising using the MSE PM losses.
{D, RGB, RGBD} Direct Vector. Processes images and uses Convolutional Neural Network to directly predict an action vector (translation and axis-angle) and supervises with MSE. The inputs are either a depth image (D), the RGB image, or an RGBD image.
(D+S, RGB+S, RGBD+S} Direct Vector. These are the same as the prior set of methods, except that the input images have additional channels corresponding to binary segmentation masks. We denote these new input images as: D+S, RGB+S, and RGBD+S. We include these baselines for fairer comparison due assuming segmentation information in the point cloud observations.
4.1.2 Experiment Protocol and Evaluation
For each task, we use a scripted demonstrator to generate a fixed set of training demos and keep the successful ones for Behavioral Cloning. We standardize on 500 training epochs for all methods and average across 5 seeds. We evaluate every 25 training epochs on 25 testing configurations and use the maximum success (averaged over 5 seeds) across the full training history, then divide this by the demonstrator success rate get the normalized performance. See Appendix A.2.2 for more details.
4.2 Simulation Results and Analysis
The results in Table 1 suggest that using ToolFlowNet outperforms the baselines on average across the tasks. In particular, for ScoopBall 4D and PourWater 6D, it outperforms all other baselines, and

---

2
3
4
5
Figure 4: An example successful ScoopBall 4D rollout by learned FlowFlowNet Behavioral Cloning policy. We show subsampled RGB frames for visual clarity, though the policy only uses point clouds as input. For the frames, we show the policy's tool flow visualizations. The policy lowers the ladle (frames 1-3), rotates and moves it in the direction of the ball (frames 4-5), lifts the ball (frame 6) and then rotates back to the starting pose (frames 7-8). The policy's 's flow visualizations for frames 2 and suggest the ability to learn downward and rotation movement, respectively. The predicted flow vectors, colored red, slightly enlarged for clarity.
for ScoopBall 6D and PourWater 3D, it is on par with the best image-based baselines. This may indicate that some tasks have a 3D nature which makes it more natural to learn policies from point clouds. Figure shows qualitative example test-time rollout of ScoopBall 4D from the learned ToolFlowNet policy. Figure also visualizes the policy's internal flow predictions (i.e., the per- point flow vectors f(()) before the SVD layer), showing that the network has learned surprisingly clean per-point tool flow vectors. Furthermore, the agent controls the ladle at its upper tip, when rotating, the flow vectors also correctly predict longer flow vectors for the points located further away from the origin of the tool pose.
4.3 Why Does ToolFlowNet Help Robot Learning?
We perform further experiments determine why ToolFlowNet outperforms the baselines that di- rectly regress transformation. Specifically, create variant of ScoopBall in which the action space consists of translations only (no rotations); see Appendix B.1 for details. These experiments reveal that ToolFlowNet does not outperform the baselines in translation-only settings, indicating that the benefits of ToolFlowNet come from predicting rotations. We also test Direct Vector methods with 4D (quaternions), 6D [60], 9D (rotation matrices) [54], and 10D [61] rotation representations Appendix 12, and find that ToolFlowNet continues to obtain higher success rates.
We next study ablations of ToolFlowNet understand which components most critical:
ToolFlowNet, No Skip Connections: removes skip connections in the segmentation PointNet++. ToolFlowNet, MSE after SVD: tests applying an MSE loss on the induced transformation from
SVD instead of point matching. We still use the consistency loss (Eq. 4).
ToolFlowNet, Point Matching (PM) Before SVD: tests using the PM loss (Eq. 3) before the
SVD layer, so the loss does brokaproe through the SVD layer.
ToolFlowNet, No Consistency: tests removing the consistency loss (and just using Eq. 3).
We use the same experiment protocol as in Section 4.1.2 on all tasks. The results, also in Ta- ble 1, suggest strong benefits to using the point matching loss, the consistency loss, and the standard segmentation PointNet++ with skip connections. For example, across all tasks, ToolFlowNet perfor- mance is worse without using a consistency loss. The utility of some design choices may be more task-specific; removing skip connections leads no successes on PourWater because removing made the model unable to predict any rotations (see Appendix B.4 for additional analysis), while it is possible to succeed in ScoopBall without using rotations.

---

Azure Kineet
(A)
3D Model
(B)
(C)
Sawyer
Target
Ludle
Tool
Flow
X
Figure 5: Physical experiments. (A) The Sawyer holds a ladle above small box with water, which is enclosed larger gray box contain spills. (B) The scanned 3D model of the ladle with representative tool flow vi- sualization. (C) Example test-tine rest-time trials with subsampled frames. Top row: successful movement towards and lifting the target. Bottom row: collision failure due to repeatedly pushing against the container.
4.4 Physical Scooping Experiments
We test ToolFlowNet in the real world using a Sawyer robot with a standard consumer ladle which we sean to obtain 3D model, and a yellow ping-pong ball acting as the target item (see Figure 5). obtain tool flow by record-
The ladle attached the Sawyer's end-effector. As in simulation, ing the change in end-effector pose and applying the transformations the tool point cloud. A Microsoft Azure Kinect camera captures top-down depth images to compute the ball's point cloud.
human operator manually moves the Sawyer's arm via direct touch to collect 125 demonstrations, with each comprising about 20 observation-actionn pairs. We use 100 demonstrations for training ToolFlowNet, and the remaining 25 for monitoring evaluation MSE. We perform 50 physical scooping trials, where each trial be- gins with the human dropping the ping- pong ball over the water at an arbitrary location within the inner box shown in Figure 5. The trial is classified as successful if the Sawyer raises the ball from the water surface to above the top of the smaller box. Results in Table suggest that the Sawyer achieves 41/50 successes (82%), with failures. All failures were due to the ladle colliding with the small box. See Appendix C for more details. future work we will do physical experiments with complex demonstrations.
ToolFoowNet in Real
#Triils
Successes Failures
41/50
9/50
Table 2: Physical scooping results.
4.5 Limitations and Failure Cases
In our experiments, obtain the ground-truth tool flow data by applying the demonstrator's 's actions on a set of tool points and computing the change in the resulting tool point positions. The tool points can be observed or derived via tool model. In either case, we require access to the demonstrator's action, and future work could relax this assumption by extracting tool flow without explicit actions, such as when a human provides a video. Possible approaches include using scene flow techniques.
A limitation of ToolFlowNet is that it may be susceptible occlusions of the tool when model of the tool is not available. For physical scooping, rely a scanned model of the tool because the Sawyer's 's arm would occlude parts of the tool, but tool models might not always be available. In future work, we will explore ways to address occlusions such as point cloud inpainting and tracking. Finally, we test ToolFlowNet on two simulation tasks with two action spaces for each, and scooping real. We hope to test on more diverse tasks such as cloth or rope manipulation, and address failures from the physical experiments.
Conclusion
In this work, we propose a general technique for policy learning from point clouds for tooe-based manipulation tasks, which demonstrate on scooping and pouring tasks. Our method, called ToolFlowNet. predicts per-point tool flow vectors which are converted into actions. We hope this research inspires future work on learning from point clouds.

---

Acknowledgments
International Conference on This work was supported by LG Electronics and by NSF CAREER grant IIS-2046491. We thank Brian Okom and Chuer Pan for assistance with the differentiable SVD layer, and Mansi Agrawal, Sashank Tirumala, and Thomas Weng for paper writing feedback.
References
[1] Mahler, Madl, V. Satish, M. Danielczuk, B. DeRose, McKinley, and K. Goldberg. Learning Ambidextrous Robot Grasping Policies, Science Robotics, 4(26), 2019.
[2] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray. J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning Dexterous In-Hand Manipulation. In International Journal of Robotics Research (IJRR), 2019.
[3] A. Zeng, P. Florence, Tompson, S. Welker, Chien, M. Attarian, T. Armstrong. I. Krasin, Duong, V. Sindhwani, and J. Lee. Transporter Networks: Rearranging the Visual World for Robotic Manipulation. In Conference on Robot Learning (CoRL), 2020.
[4] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. TossingBot: Learning to Throw Arbitrary Objects with Residual Physics. In Robotics: Science and Systems (RSS), 2019.
[5] S. Levine, C. Finn, T. Darrell, and Abbeel. End-to-end Training of Deep Visuomotor Policies. In Journal of Machine Learning Research (JMLR), 2016.
[6] L. Pinto and Gupta. Supersizing Sell-supervision: Learning to Grasp from S0K Tries and 700 Robot Hours. In IEEE International Conference on Robotics and Automation (ICRA), 2016.
(7] OpenAT, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plap- pert, G. Powell, R. Ribas, J. Schneider. N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan. W. Zaremba, and L. Zhang. Solving Rubik's Cube with a Robot Hand. arXiv preprint arXiv: 191001910.07113, 2019.
[8] T. Haarnoja, Zhou, K. Hartikainen, Tucker, Ha, Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft Actor-Critic Algorithms and Applications. arXiv preprint arXiv: 1rXiv:1812.05900 2018.
[9] N. Jakobi, P.Husbands, and L. Harvey. Noise and the Reality Gap: The use of Simulation in Evolutionary Robotics In European Conference on Advances in Artificial Life, 1995.
[10] O. Kroemer, S. Niekum, and G. Konidaris. A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms, arXiv preprint 190Xx::900.000 03 146, 2019.
[11] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar. Robotic Manipulation and Sensing of Deformable Objects in Domestic and Industrial Applications: a Survey. In International Journal of Robotics Research (IJRR). 2018.
[12] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon F. Alambcigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, J. Pan, W. Yuan, and M. Gienger. Challenges and Outlook in Robotic Manipulation of Deformable Objects. arXiv preprint arXiv:2105.01767, 2021.
[13] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet:
Deep Learning on Point Sets for 3D Classification and Segmentation. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[14] H. Zhao, L. Jiang, J. Jia, P. Torr, and Koltun. PointTransformer.
Computer Vision (ICCV), 2021.
[15] C. R. Qi, Yi, H. Su, and L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning Point Sets in Metric Space. In Neural Information Processing Systems (NeurIPS), 2017.
[16] Y. Zhou and O. Tuzel. VoxeINet: End-to-Fad Learning for Point Cloud Based 3D Object Detection. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[17] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics (TOG), 2019.
[18] T. Chen, J. Xu, and P. Agrawal. A System for General In- Hand Object Re-Orientation. In Conference on Robot Learning (CoRL), 2021.

---

[19] L. Wang, Y. Xiang, W. Yang. A. Mousavian, and D. Fox. Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds. In Conference Robot Learning (CoRL), 2021.
[20] S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade. Three-dimensional Scene Flow, In International Conference Computer Vision (ICCV), 1999.
[21] Z. Liu, H. Tang, Y. Lin, and S. Han. Point-Voxel CNN for Efficient 3D Deep Learning. In Neural Information Processing Systems (NewrIPS), 2019.
[22] W. Wu, Z. Qi, and L. Fuxin. PointConv: Deep Convolutional Networks on 3D Point Clouds. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2019.
[23] X. Liu, M. Yan, and J. Bohg. MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences. In International Conference on Computer Vision (ICCV), 2019.
[24] B. K. Horn and B. G. Schunck, Determining Optical Flow. Technical report, USA, 1980.
[25] P. Fischer, A. Dosovitskiy, E. 1lg, P. Hausser, C. Hazrbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. FlowNet: Learning Optical Flow with Convolutional Networks. International Conference on Computer Vision (ICCV), 2015.
[26] 7. Teed and Deng. RAFT-3D: Scene Flow using Rigid-Mation Embeddings. In Conference on Com- puter Vision and Puttern Recognition (CVPR), 2021.
[27] Y.H. Wu, J. Wang, and X. Wang. Learning Generalizable Dexterous Manipulation from Human Grasp Affordance. In Conference on Robot Learning (CoRL), 2022.
[28] K. Lobos- Tsunekawa and T. Harada. Point Cloud Based Reinforeement Learning for Sim-to- Real and Partial Observability in Visual Navigation. In IEEE/RSI International Conference Intelligent Robots and Systems (IROS), 2020.
[29] B. Thach, B. Y. Cho. A. Kuntz, and T. Hermans. Learning Visual Shape Control of Novel 3D Deformable Objects from Partial- Point Clouds. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[30] 7. Qin, K. Fang, Zhu, I.. Fei-Fei, and Savarese. KETO: Learning Keypoint Representations for Tool Manipulation. JEEE International Conference on Rohotics and Automation (ICRA), 2020.
[31] T. Weng. S. Bajracharya. Y. Wang, K. Agrawal, and D. Held. Fawric Bimanual Cloth Manipula- tion with Flow-based Policy. In Conference on Robot Learning (CoRL), 2021.
[32] S. Pillai, M. R. Walter, and S. Teller. Learning Articulated Motions From Visual Demonstration. In Robotics: Science and Systems (RSS), 2014.
[33] B. Eisner, H. Zhang, and D. Held. FlowBor3D: Learning 3D Articulation Flow to Manipulate Articulated Objects. In Robotics: Science and Systems (RSS), 2022.
[34] B. Shen, Z. Jiang, C. Choy, L. J. Guibas, S. Savarese, A. Anandkumar, and Y. Zhu. ACID: Action- Conditional Implicit Visual Dynamies for Deformable Object Manipulation. In Robotics: Science and Systemy (RSS), 2022.
[35] A. Goyal, A. Mousavian, C. Paxton, Y.-W. Chao, B. Okom, J. Deng, and D. Fox. IFOR: Iterative Flow Minimization for Robotic Object Rearrangement. In Conference Computer Vision and Puttern Recog- nition (CVPR), 2022.
[36] S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez. Tactile-RI for Insertion: Generalization to Objects of Unknown Geometry. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[37] M. Argus, L. Hermann, J. Long, and T. Brox. FlowControl: Optical Flow Based Visual Servoing. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[38] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Péres Learning Compositional Models of Robot Skills for Task and Motion Planning. In International Journal of Robotics Research (IJRR), 2019.
[39] C. Schenck, J. Tompson, D. Fox, and S. Levine. Learning Robotic Manipulation of Granular Media. In Conference Robot Learning (CoRL), 2017.
[40] C. Schenck and D. Fox. Towards Learning to Perceive and Reason About Liquids. In International Symposium on Experimental Robotics (ISER), 2016.
10

---

[41] C. Schenck and D. Fox. Visual Closed-Loop Control for Pouring Liquids. In IEEE International Confer- on Robotics and Automation (ICRA), 2017.
[42] C. Schenck and D. Fox. SPNets: Differentiable Fluid Dynamics for Deep Neural Networks, In Conference on Robot Learning (CoRL., 2018.
[43] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer: Learning Audio Feedback for Estimating Amount and Flow of Granular Material. In Conference on Robot Learning (CoRL), 2018.
[44] C. Matl, Y. Narang, R. Bajesy, F. Ramos, and D. Fox. Inferring the Material Properties of Granular Media for Robotic Tasks. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[45] G. Salhotra, T.-C. A. Liu, M. Dominguez-Kuhann and G. Sukhatme. Learning Deformable Ohject Manipulation from Expert Demonstrations. In IEEE Robotics and Automation Letters (RA-1J), 2022.
[46] D. A. Pomerleau. Efficient Training Artificial Neural Networks for Autonomous Navigation. Neural Comput., 3, 1991.
[47] S. Ross, G. Gordon, and A. Bagnell. A Reduction of Imitation Learning and Structured Predic- tion to No-Reeoee No-Regret Online Learning. In International Conference on Artificial Intellignnce and Statistics (AISTATS), 2011.
[48] S. Dasari, Wang. J. Hong, Bahl, Y. Lin, A. Wang, A. Thankaraj. K. Chahal, B. Calli, S, Gupta, D. Held, L. Pinto. D. Pathak, V. Kumar, and A. Gupta. RB2: Robotic Manipulation Benchmarking with Twist. NeurlPS 2021 Datasets and Benchmarks Track, 2021.
[49] P. Florence, C. Lynch, A. Zeng. O. Ramirez, Wahid, L.. Downs, A. Wong, Lee, I. Mordatch, and
J. Tompson. Implicit Behavioral Cloning. In Conference on Robot Learning (CoRL.), 2021.
[50] E. Johns. Coarse to-Fiee Imitation Learning: Robot Manipulation from a Single Demonstration. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[51] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, Abbeel, and J. Peters. An Algorithmic Perspective on Imitation Learning. Foundations and Trends in Robotics, 7, 2018.
[52] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A Survey of Rohot Learning From Demonstra- tion. Robotics and Autonomous Systems, 57, 2009.
[53] O. Sorkine-Homung Hornung and M. Rabinovich. Least- heast-Squares Rigid Motion Using SVD, Technical report, ETH Zurich, 2016.
[54] J. Levinson, C. Esteves, K. Chen, N. Snavely, A. Kanazawa, A. Rostamizadeh, and A. Makadia. An Analysis of SVD for Deep Rotation Estimation. In Neural Information Processing Systems (NeurIPS), 2020.
[55] Y. Xiang. T. Schmidt, V. Narayanan, and D. Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Robotics: Science and Systems (RSS), 2018
[56] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox. DeepIM: Deep Iterative Matching for 6D Pose Estimation. In European Conference Computer Vision (ECCV), 2018.
[57] C. Wang, D. Xu, Y. Zhu, R. Martín-Martin, C. Lu, L. Fei-Fei, and S. Savarese. DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[58] X. Lin, Y. Wang, J. Olkin, and D. Held. SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation. In Conference on Robot Learning (CoRL), 2020.
[59] M. Macklin, M. Muller, N. Chentanez, and T.-Y. Kim. Unified Particle Physics for Real-Time Applica- tions. ACM Trans. Graph., 33(4). July 2014.
[60] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the Continuity of Rotation Representations in Neural Networks. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[61] V. Peretroukhin, M. Giamou, D. M. Rosen, W. N. Greene, N. Roy, and J. Kelly. A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty. In Robotics: Science and Systems (RSS), 2020.
[62] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015.
11

---

[63] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to Rearrange Deformable Cables, Fabrics, and Bags with Coal-Conditionse Transporter Networks. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[64] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep Reinforcement Learning that Matters. Association for the Advancement of Artificial Intelligence (AAAI), 2018.
[65] A. Srinivas, M. Laskin, and P. Abbeel. CURL: Contrastive Unsupervised Representations for Reinforce- ment Learning. In International Conference on Machine Learning (ICML), 2020.
[66] M. Fey and J. E. Lenssen. Fast graph representation learning with Py Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[67] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Accelerating 3D Deep Learning with PyTorch3. (rXiv:2007.08500 2020.
[68] J. Chen, Y. Yin, T. Birdal, B. Chen, L. Guibas, and H. Wang. Projective Manifold Gradient Layer for Deep Rotation Regression. Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[69] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, Sendonaris, G. Dulac-Amold. Osband, Agapiou, Leibo, and A. Gruslys. Deep Q-learning from Demonstra- tions. In Association for the Advancement of Artificial Intelligence (AAAI), 2018.
[70] A. Nair, McGrew, M. Andrychowicz, Zaremba, and P. Abbeel. Overcoming Exploration in Rein- forcement Learning with Demonstrations. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[71] L. Rozo and V. Dave. Orientation Probabilistic Movement Primitives on Riemannian Manifolds. In Conference Robot Learning (CoRL), 2021.
[72] P. Pastor, L. Righetti, M. Kalakrishnan, and S. Schaal. Online Movement Adaptation Based on Previous Sensor Experiences. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011.
[73] J. Urain, D. Tateo, and J. Peters. Learning Stable Vector Fields on Lie Groups. In IEEE Rohotics and Automation Letters (RA-L), 2021.
[74] Y. Huang, F. J. Abu-Dakka. J. Silverio, and D. G. Caldwell. Toward Orientation Learning and Adaptation in Cartesian Space. In IEEE Transactions Robotics, 2020.
[75] M. Liu, X. Li, Z. Ling, Li, and H. Su. Frame Mining: Free Lunch for Learning Robotic Manipulation from 3D Point Clouds. In Conference Robot Learning (CoRL), 2022.
[76] Zeromq. URL bettps://zerog: org/.
12


Reviews:
[{"review_id": "zGAf-YqTcb", "reviewer": "Reviewer_LESm", "summary": "This paper presents a framework for learning robotic manipulation tasks with tools using point clouds. To extract point cloud observations, it utilizes a dense point cloud network. It proposes a point-cloud-to-action system, ToolFlowNet, to generate a full transformation matrix to estimate the motion of point clouds. An algorithmic demonstrator along with the ToolFlowNet is used to generate a dataset for training a behavior cloning policy.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The method can be easily implemented and used for any Learning from Demonstration (LfD) methods.  \n* The approach is demonstrated on a robotic system.\n* Figures are clear and useful for understanding the method and results.\n\n**Weaknesses:**\n* The main problem with this paper is the lack of comparison between the proposed method and state-of-the-art point-cloud-based or non-point-cloud-based methods. Moreover, most baselines are variants of your method (Direct Vector, Dense Transformation) or generic methods (Image CNN), and they are too weak to be considered competitive methods. A competitive point-cloud-based method could be [1]. Some competitive non-point-cloud-based methods include SAC, SAC-DrQ, SAC-CURL from Softgym, or a recent LfD method implemented in Softgym [2].\n* The evaluation metric (success rate / success rate of the demonstrator) used in the experiments may not be suitable. It is possible for reinforcement learning (RL) and BC+RL methods to exceed the demonstrator’s performance. Even though the experiments do not contain these methods (which I think they should), it is better to show the unnormalized success rate in experiments. \n* Behavior Cloning (BC) is known to suffer from the distributional shift problem. Since BC is utilized in this framework, how do you deal with this problem?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak reject in light of the \"strengths and weaknesses\" discussion above. The main issue of this paper is that it is not clear how this method compares to state-of-the-art point-cloud-based methods and non-point-cloud-based methods. Due to the lack of this comparison, it is not possible to draw the conclusion that learning from point clouds using ToolFlowNet is superior to learning from RGB or depth images.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Introduction:\n* Consider re-writing the first sentence (lines 17 to 19). The first sentence of the introduction should be clear and concise.\n\nMethod:\n* [3.2] It’s not clear why the ground truth transformation is not used in the consistency loss. Consider adding this information on line 154.\n\nExperiments:\n* Table 1: Why does “ToolFlowNet, No Skip Conn.” have a success of 0% in the PourWater task?  How would skip connections help when learning to predict rotations? The authors should explain this more clearly (lines 243 - 245).\n* It is a good idea to show the success rate of the demonstration data in Table 1. \n\n\n**References:**\n\n[1] Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, and Yuke Zhu. ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation, 2022.\n\n[2] Gautam Salhotra, I-Chun Arthur Liu, Marcus Dominguez-Kuhne, and Gaurav S. Sukhatme. Learning Deformable Object Manipulation from Expert Demonstrations, 2022.\n"}}, {"review_id": "u2ZGamwBjX", "reviewer": "Reviewer_Vzsj", "summary": "The paper proposes ToolFlowNet, a novel point-cloud-based architecture that predicts the per-point tool flow vectors. Those vectors are used to formulate an SE(3) transformation as an action for the robot to execute. Experimental evaluation is performed in both the simulation and the real world with convincing results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of using per-point flow vectors to formulate an action is novel and interesting. Though [33] used a similar idea to predict articulation flow for manipulating articulated objects, this paper proposes a novel perspective to use flow vector to control an arbitrary tool.\n2. The paper is well-written and easy to follow.\n\nWeaknesses:\n\n1. The Image CNN baseline seems unfair because it uses RGB images instead of RGBD images or depth images while the proposed method uses point cloud that contains depth information.\n2. Although the method is defined in SE(3), the action space in the experiments is limited in 3D ($\\Delta x$, $\\Delta y$, $\\Delta \\theta$) or 4D ($\\Delta x$, $\\Delta y$, $\\Delta z$, $\\Delta \\theta$) instead of 6D. The limitation section does not discuss why 6D task is not demonstrated and the potential issues when using this method in a 6D action space.\n3. Showing two tasks in Figure 2 is a bit misleading. I thought the module takes in two separate point clouds at first glance. I would recommend only keeping one task in this figure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel approach to predict the per-point flow using a point net architecture, then combine the per-point flow on a tool to generate an SE(3) action of the end-effector. Though the paper has some minor weaknesses (see above), I think it is a good contribution that provides new insight on how to use point cloud and point net architectures in robotic manipulation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The Image CNN baselines only process RGB information but not depth information, which seems a bit unfair.\n\nA minor question: in the real-world experiment, how do you obtain the segmentation labels?", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "UOZR9OnD0F", "reviewer": "Reviewer_sFXa", "summary": "The paper proposes a method to learn behavioral cloning motion policies for arbitrary tools, given a point cloud.  In particular, the network introduces a network architecture to do so.\n\nThe main contributions of the paper are:\n1. A network architecture that learns the tool’s motion policies through the learning of the point cloud flow. The authors claim that estimating \n    the tool’s point cloud flow F prior to the tool’s transformation T leads to improved performance with respect to estimating the tool’s \n    transformation directly.\n2. The paper is evaluated in two simulated tasks. Learning how to pour water from a box and learning how to scoop a ball with a ladle.\n3. A real robot experiment in which the scoop of a ball is learned from real human demonstrations.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To the reviewer's view, the main strengths of the paper are:\n\n1. A fully differentiable architecture that outputs the translation vector and rotation matrix of the tool, through the tool’s point cloud flow. I was not aware that it was possible to fully differentiate the whole pipeline.\n\n2. Considering the whole scene point cloud as input observation data to learn the tool’s motion policy.\n\n3. If sufficiently validated, the claim that “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”.\n\nTo the reviewer's view, the main weaknesses of the paper are:\n\n1. Too simple experiments. There is no experiment considering 6D movements. The chosen experiments assume: 2D translation + 1D rotation in the pouring experiment, 3D translation + 1D rotation in the ball scooping and 3D translation for the physical scooping experiment. To properly validated the proposed method, it is essential to show its performance in a 6D motion generation task. If a motion is not exactly aligned with the world frame, the motion representation is going to be decoupled along multiple axis and thus, highly beneficial to show that the system is able to learn these motions both with simulated and real data.\n\n2. The paper misses a proper analysis on why  “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”. It is not intuitive why learning the flow of a whole pointcloud should enhance the performance. Previous works on learning behavioral cloning motion policies on SE(3) [1, 2, 3, 4] directly learn the motion policy in SE(3), without the pointcloud flow. These works assume access to the object’s 6D pose and learn the change in the pose. It would be relevant to observe if the pointcloud flow based motion representation is actually better than first computing the 6D pose of the object and then, estimating the transformation given the pose. This would help understanding the benefit of pointcloud flow.\n\n3. The network is trained using the flow of the tool’s pointcloud. Therefore, the data is required to store a one-to-one mapping between all the points in the pointcloud. Yet, when recording the pointcloud with and RGB-D camera, this information is missing or some regions of the tool might be occluded in some frames. It would be interesting to know if the method is limited to pointcloud data that has been generated synthetically or if the authors have any proposition on how to extend the method to raw RGB-D training data in which the alignment between the points is missing. If the solution is to estimate first the pose, then, could we use this pose as proposed in weakness 2? \n\n[1] Pastor, Peter, et al. \"Online movement adaptation based on previous sensor experiences.\" 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2011.\n\n[2] Huang, Yanlong, et al. \"Toward orientation learning and adaptation in cartesian space.\" IEEE Transactions on Robotics 37.1 (2020): 82-98.\n\n[3] Rozo, Leonel, and Vedant Dave. \"Orientation Probabilistic Movement Primitives on Riemannian Manifolds.\" Conference on Robot Learning. PMLR, 2022.\n\n[4] Urain, Julen, Davide Tateo, and Jan Peters. \"Learning Stable Vector Fields on Lie Groups.\" arXiv preprint arXiv:2110.11774 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an idea that could be interesting for learning motion policies from point clouds. Nevertheless, the work is missing a proper experimental validation and a proper comparison with respect to classic methods (Compute the pose of the object and learn the motion given the known pose).\n\nThe authors need to show the performance of their proposed method for more complex tasks that require 6D motion.\n\nThe authors need to show the difference in using the point cloud flow with respect to estimating the tools pose and using this state information to learn the motion policy (classic approach) and thus, validate their claim that “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”.\n\nThe work would also benefit from an experiment showing the performance of the method with real RGB-D data instead of synthetic one.\n\nDue to these limitations, even if the work has potential, I suggest a weak rejection of the paper until the arisen problems are tackled.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Major issues has been previously introduced.\n\nMinor:\nIn line 123, T should be bold.\n\n\n", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "TS_xZvUtdsL", "reviewer": "Reviewer_ckzp", "summary": "This paper resents a method in which robust policies can be learned using only few demonstrations. It does this on the basis of point cloud inputs and a network that outputs the predicted future optical flow according from a set of given scripted demonstrations. The paper includes extensive baseline experiments in combination with real world experiments. The paper is well written but the method is difficult to understand.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Presents a method in which robust policies can be learned using only few demonstrations.\n2. The paper combines a relatively extensive ablation study for simulated results\n3. The paper performs real works experiments.\n\nWeaknesses:\n\n1. The method is not easily understood.\n2. Some references that should be included are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think that the paper presents a good method, with good experiments. However it seems to be hastily written and is still lacking in clarity, substantial revisions should be made before acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Major Points:\nThe method is not explained in a very clear way. My current understanding is as follows:\n\na) The policy a = \\pi(o) computes the dense flow F as an intermediate step, so a = \\pi ( F(o) ). This is, however, not the flow in the classical sense t_{-1} -> t_0, but the predicted flow into the future. From F we then extract T, which we convert into a:  a = \\pi(T(F_\\theta(o))). The network producing F is then supervised using only a. An explicit evaluation of F, on new inputs, is not possible, since it depends not only on the tool positions, but also the manipulated object e.g. yellow ball position. \n\na2) I was then confused when I read the limitations section, I initially thought ground truth tool flow refers to F, but it probably means T.\n\nb) For the In the DirectVector MSE baseline its not clear if MSE is applied to the actions directly or the points, both variants should be included.\n\nc) The notation is a bit unclear. Input point clouds are always referred to as p, it is introduced as *segmented* point clouds. I am assuming that the point clouds required during the runtime of the algorithm do not need to be segmented. This is a bit confusing in combination with the references to \"classification\" and \"segmentation\" PointNets in L195, L199, as L195 seems to mean an action regression network and L199 a per-point regression network. The potential confusion is wrt. discrete action classification and tool mask segmentation.\n\nWhy is the method difficult to understand:\na) Someone coming from an optical flow or scene flow background would expect a ToolFlowNet to output F, and a network to be trained and evaluated on this task. This difference should be more explicitly highlighted.\n\nb) Understanding what the method did required piecing together information from different sections, more explicit formulas like a = \\pi(T(F_\\theta(o))) would help.\n\nc) Some variables in Figure 2. would also help.\n\nA reference to the \"FlowControl: Optical Flow Based Visual Servoing\" should be included, as indicated by the name many aspects of the paper are similar to this work, also addressing the topic of robot manipulation using segmented RGB-D inputs to compute MSE between point clouds to find relative transformations for robot motion.  Maybe also \"Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration\"\n\nMinor Points:\n\nWhy not include color information in the point clouds, you should get this for \"free\" in the simulation as well as your experimental setup.\n\nThe Point matching loss solves a similar objective to the MSE loss, having these appear explicit side by side would clarify the differences.\n\nIn line 115 there should probably be a P'_{tool}, \n\nLine 115/120 seem to be a bit redundant, maybe it could be reformulated so that T*,F_gt,F,a*,a^ are explained compactly as these are  similar."}}]

Author rebuttal:

Decision: Accept (poster)