Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a framework for learning robotic manipulation tasks with tools using point clouds. To extract point cloud observations, it utilizes a dense point cloud network. It proposes a point-cloud-to-action system, ToolFlowNet, to generate a full transformation matrix to estimate the motion of point clouds. An algorithmic demonstrator along with the ToolFlowNet is used to generate a dataset for training a behavior cloning policy.  
strengths_and_weaknesses: **Strengths:**
* The method can be easily implemented and used for any Learning from Demonstration (LfD) methods.  
* The approach is demonstrated on a robotic system.
* Figures are clear and useful for understanding the method and results.

**Weaknesses:**
* The main problem with this paper is the lack of comparison between the proposed method and state-of-the-art point-cloud-based or non-point-cloud-based methods. Moreover, most baselines are variants of your method (Direct Vector, Dense Transformation) or generic methods (Image CNN), and they are too weak to be considered competitive methods. A competitive point-cloud-based method could be [1]. Some competitive non-point-cloud-based methods include SAC, SAC-DrQ, SAC-CURL from Softgym, or a recent LfD method implemented in Softgym [2].
* The evaluation metric (success rate / success rate of the demonstrator) used in the experiments may not be suitable. It is possible for reinforcement learning (RL) and BC+RL methods to exceed the demonstrator’s performance. Even though the experiments do not contain these methods (which I think they should), it is better to show the unnormalized success rate in experiments. 
* Behavior Cloning (BC) is known to suffer from the distributional shift problem. Since BC is utilized in this framework, how do you deal with this problem?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak reject in light of the "strengths and weaknesses" discussion above. The main issue of this paper is that it is not clear how this method compares to state-of-the-art point-cloud-based methods and non-point-cloud-based methods. Due to the lack of this comparison, it is not possible to draw the conclusion that learning from point clouds using ToolFlowNet is superior to learning from RGB or depth images.
issues: Introduction:
* Consider re-writing the first sentence (lines 17 to 19). The first sentence of the introduction should be clear and concise.

Method:
* [3.2] It’s not clear why the ground truth transformation is not used in the consistency loss. Consider adding this information on line 154.

Experiments:
* Table 1: Why does “ToolFlowNet, No Skip Conn.” have a success of 0% in the PourWater task?  How would skip connections help when learning to predict rotations? The authors should explain this more clearly (lines 243 - 245).
* It is a good idea to show the success rate of the demonstration data in Table 1. 


**References:**

[1] Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, and Yuke Zhu. ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation, 2022.

[2] Gautam Salhotra, I-Chun Arthur Liu, Marcus Dominguez-Kuhne, and Gaurav S. Sukhatme. Learning Deformable Object Manipulation from Expert Demonstrations, 2022.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to learn behavioral cloning motion policies for arbitrary tools, given a point cloud.  In particular, the network introduces a network architecture to do so.

The main contributions of the paper are:
1. A network architecture that learns the tool’s motion policies through the learning of the point cloud flow. The authors claim that estimating 
    the tool’s point cloud flow F prior to the tool’s transformation T leads to improved performance with respect to estimating the tool’s 
    transformation directly.
2. The paper is evaluated in two simulated tasks. Learning how to pour water from a box and learning how to scoop a ball with a ladle.
3. A real robot experiment in which the scoop of a ball is learned from real human demonstrations.

strengths_and_weaknesses: To the reviewer's view, the main strengths of the paper are:

1. A fully differentiable architecture that outputs the translation vector and rotation matrix of the tool, through the tool’s point cloud flow. I was not aware that it was possible to fully differentiate the whole pipeline.

2. Considering the whole scene point cloud as input observation data to learn the tool’s motion policy.

3. If sufficiently validated, the claim that “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”.

To the reviewer's view, the main weaknesses of the paper are:

1. Too simple experiments. There is no experiment considering 6D movements. The chosen experiments assume: 2D translation + 1D rotation in the pouring experiment, 3D translation + 1D rotation in the ball scooping and 3D translation for the physical scooping experiment. To properly validated the proposed method, it is essential to show its performance in a 6D motion generation task. If a motion is not exactly aligned with the world frame, the motion representation is going to be decoupled along multiple axis and thus, highly beneficial to show that the system is able to learn these motions both with simulated and real data.

2. The paper misses a proper analysis on why  “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”. It is not intuitive why learning the flow of a whole pointcloud should enhance the performance. Previous works on learning behavioral cloning motion policies on SE(3) [1, 2, 3, 4] directly learn the motion policy in SE(3), without the pointcloud flow. These works assume access to the object’s 6D pose and learn the change in the pose. It would be relevant to observe if the pointcloud flow based motion representation is actually better than first computing the 6D pose of the object and then, estimating the transformation given the pose. This would help understanding the benefit of pointcloud flow.

3. The network is trained using the flow of the tool’s pointcloud. Therefore, the data is required to store a one-to-one mapping between all the points in the pointcloud. Yet, when recording the pointcloud with and RGB-D camera, this information is missing or some regions of the tool might be occluded in some frames. It would be interesting to know if the method is limited to pointcloud data that has been generated synthetically or if the authors have any proposition on how to extend the method to raw RGB-D training data in which the alignment between the points is missing. If the solution is to estimate first the pose, then, could we use this pose as proposed in weakness 2? 

[1] Pastor, Peter, et al. "Online movement adaptation based on previous sensor experiences." 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2011.

[2] Huang, Yanlong, et al. "Toward orientation learning and adaptation in cartesian space." IEEE Transactions on Robotics 37.1 (2020): 82-98.

[3] Rozo, Leonel, and Vedant Dave. "Orientation Probabilistic Movement Primitives on Riemannian Manifolds." Conference on Robot Learning. PMLR, 2022.

[4] Urain, Julen, Davide Tateo, and Jan Peters. "Learning Stable Vector Fields on Lie Groups." arXiv preprint arXiv:2110.11774 (2021).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper presents an idea that could be interesting for learning motion policies from point clouds. Nevertheless, the work is missing a proper experimental validation and a proper comparison with respect to classic methods (Compute the pose of the object and learn the motion given the known pose).

The authors need to show the performance of their proposed method for more complex tasks that require 6D motion.

The authors need to show the difference in using the point cloud flow with respect to estimating the tools pose and using this state information to learn the motion policy (classic approach) and thus, validate their claim that “estimating the tool flow leads to improved performance compared to estimating the transformation T directly”.

The work would also benefit from an experiment showing the performance of the method with real RGB-D data instead of synthetic one.

Due to these limitations, even if the work has potential, I suggest a weak rejection of the paper until the arisen problems are tackled.

issues: Major issues has been previously introduced.

Minor:
In line 123, T should be bold.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper resents a method in which robust policies can be learned using only few demonstrations. It does this on the basis of point cloud inputs and a network that outputs the predicted future optical flow according from a set of given scripted demonstrations. The paper includes extensive baseline experiments in combination with real world experiments. The paper is well written but the method is difficult to understand.
strengths_and_weaknesses: Strengths:

1. Presents a method in which robust policies can be learned using only few demonstrations.
2. The paper combines a relatively extensive ablation study for simulated results
3. The paper performs real works experiments.

Weaknesses:

1. The method is not easily understood.
2. Some references that should be included are missing.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I think that the paper presents a good method, with good experiments. However it seems to be hastily written and is still lacking in clarity, substantial revisions should be made before acceptance.
issues: Major Points:
The method is not explained in a very clear way. My current understanding is as follows:

a) The policy a = \pi(o) computes the dense flow F as an intermediate step, so a = \pi ( F(o) ). This is, however, not the flow in the classical sense t_{-1} -> t_0, but the predicted flow into the future. From F we then extract T, which we convert into a:  a = \pi(T(F_\theta(o))). The network producing F is then supervised using only a. An explicit evaluation of F, on new inputs, is not possible, since it depends not only on the tool positions, but also the manipulated object e.g. yellow ball position. 

a2) I was then confused when I read the limitations section, I initially thought ground truth tool flow refers to F, but it probably means T.

b) For the In the DirectVector MSE baseline its not clear if MSE is applied to the actions directly or the points, both variants should be included.

c) The notation is a bit unclear. Input point clouds are always referred to as p, it is introduced as *segmented* point clouds. I am assuming that the point clouds required during the runtime of the algorithm do not need to be segmented. This is a bit confusing in combination with the references to "classification" and "segmentation" PointNets in L195, L199, as L195 seems to mean an action regression network and L199 a per-point regression network. The potential confusion is wrt. discrete action classification and tool mask segmentation.

Why is the method difficult to understand:
a) Someone coming from an optical flow or scene flow background would expect a ToolFlowNet to output F, and a network to be trained and evaluated on this task. This difference should be more explicitly highlighted.

b) Understanding what the method did required piecing together information from different sections, more explicit formulas like a = \pi(T(F_\theta(o))) would help.

c) Some variables in Figure 2. would also help.

A reference to the "FlowControl: Optical Flow Based Visual Servoing" should be included, as indicated by the name many aspects of the paper are similar to this work, also addressing the topic of robot manipulation using segmented RGB-D inputs to compute MSE between point clouds to find relative transformations for robot motion.  Maybe also "Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration"

Minor Points:

Why not include color information in the point clouds, you should get this for "free" in the simulation as well as your experimental setup.

The Point matching loss solves a similar objective to the MSE loss, having these appear explicit side by side would clarify the differences.

In line 115 there should probably be a P'_{tool}, 

Line 115/120 seem to be a bit redundant, maybe it could be reformulated so that T*,F_gt,F,a*,a^ are explained compactly as these are  similar.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes ToolFlowNet, a novel point-cloud-based architecture that predicts the per-point tool flow vectors. Those vectors are used to formulate an SE(3) transformation as an action for the robot to execute. Experimental evaluation is performed in both the simulation and the real world with convincing results.
strengths_and_weaknesses: Strengths:

1. The idea of using per-point flow vectors to formulate an action is novel and interesting. Though [33] used a similar idea to predict articulation flow for manipulating articulated objects, this paper proposes a novel perspective to use flow vector to control an arbitrary tool.
2. The paper is well-written and easy to follow.

Weaknesses:

1. The Image CNN baseline seems unfair because it uses RGB images instead of RGBD images or depth images while the proposed method uses point cloud that contains depth information.
2. Although the method is defined in SE(3), the action space in the experiments is limited in 3D ($\Delta x$, $\Delta y$, $\Delta \theta$) or 4D ($\Delta x$, $\Delta y$, $\Delta z$, $\Delta \theta$) instead of 6D. The limitation section does not discuss why 6D task is not demonstrated and the potential issues when using this method in a 6D action space.
3. Showing two tasks in Figure 2 is a bit misleading. I thought the module takes in two separate point clouds at first glance. I would recommend only keeping one task in this figure.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a novel approach to predict the per-point flow using a point net architecture, then combine the per-point flow on a tool to generate an SE(3) action of the end-effector. Though the paper has some minor weaknesses (see above), I think it is a good contribution that provides new insight on how to use point cloud and point net architectures in robotic manipulation.
issues: The Image CNN baselines only process RGB information but not depth information, which seems a bit unfair.

A minor question: in the real-world experiment, how do you obtain the segmentation labels?
review_title: null

Meta-review: metareview: This paper presents an imitation learning approach for manipulation using tools based on the tool’s point cloud flow. The authors claim that using point cloud results in better performance than using transformation. The method is demonstrated with 2 tasks x 2 action spaces in simulation. One of the tasks is also implemented on hardware. Overall, the proposed method performs best on average, if not the best in every scenario.

The authors addressed most of the reviewer comments, including new examples with 6D action space. If the paper is accepted, the authors should clarify the role of tool pose in the training phase (cf. A2 and A3 in the response to reviewer sFXa). Also, a deeper discussion on why using point cloud is better than using transformation is appreciated given that prior works have successfully used quaternions and vector-angle representation.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)