Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a model-based contact servoing framework to perform control the contact between a compliant tool and the robot's environmet. Contact is parameterized as a binary contact flag, line of contact and an end-effector wrench. Dynamics are learned in a latent space with an encoder-decoder framework that is used to predict the contact parameters at every step using a pointcloud observation and the input wrench measured at the robot's wrist.
strengths_and_weaknesses: Strengths:
- Data collection procedure is self-supervised without needing human intervention or labelling.
- In-depth analysis of different ablations of the model to study and evaluate the effectiveness of different components of the system

Weaknesses:
- Generalization: Experiments are restricted to a single spatula. Furthermore, the surface in contact with the spatula also remains the same. Would be interesting to see generalization to new spatulas, or even just train/test on more tools as well as analyzing variations in other parameters such as the contact surface.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While it would be interesting to see the performance of the proposed framework in noisier, more realistic settings, the method is promising and the presented analysis is sound and well-executed. The paper is able to demonstrate controlled extrinsic dexterity with a deformable spatula while using an impedance controller, which is a promising direction as we start moving towards robots operating in human environments. 
issues: - It would be nice to see experiments and analysis of generalization, both in terms of the generalization of the model to other tools, environments, surfaces etc as well as the generalization of the method to more general, parameterizable contact shapes.
- One baseline that would be interesting to compare against, especially in the cases of tracing contact areas where the tool is NOT tilted, would be to apply a larger force than required and using just vision to track the line. While this would not be able to handle partial contacts, it would be interesting to see if this method can solve the non-partial contact servoing tasks (like the straight area) as well as the proposed method.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a robot control framework for controlling contact forces at the tip of a pre-grasped tool. Remarkably, the proposed approach considers a situation where the compliance of the grasped tool can be used to change the geometry of the contact. The authors presents a method to learn a dynamic model which allows to predict the effects to the robot actions on the contact geometry and on the robot end-effector location. This dynamic model is used in an MPC framework to control contact forces in a variety of scenarios, including controlling the contact forces at the tip of a flexible spatula to scrape a target object in presence of obstacles.
strengths_and_weaknesses: The paper presents a novel solution to a novel problem. Controlling tool contact forces is a very interesting problem which (to my knowledge) has been addressed only in the rigid case. The submitted paper considers the non-rigid case which is a significant novelty. The major limitation of the paper is the need of a complicated data-gathering phase which requires precision equipment (Photoneo) and human supervision. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The submitted paper presents a novel task and an interesting solution. The paper is well written and easy to follow. Videos are interesting to watch and the proposed solution is sound and technically correct.
issues: - ***Action specifications***. At line 93 author mention the action trajectory $\mathbf{a}_{0:T-1}$ but it isn't clear what is the action space used for the robot. Authors should make this clear specifying if they control torques, positions and/or impedance.

- ***Typo on line 88***. $g_{i}+1$ should be $g_{i+1}$.

- ***Typo on line 81***. $\mathbb{R}^{2,3}$ should be $\mathbb{R}^{2 \times 3}$.

- ***Improve notation***. Since $G = \[ g_1, \dots, g_i, \dots, g_L\]$ are trajectories in time I suggest the notation $G = \[ g_1, \dots, g_t, \dots, g_T\]$.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors proposed a learning approach for modeling tool-environment interaction which learn the dynamics from real world data.	
Key of proposed method is embedding the latent spaces from the robot's sensor data and decodes the contact feature representation which consists of binary contact value, the line of contact, and an end-effector wrench.	The authors verified the scraping operation of a clay-like object using a spatula attached to the tip of an actual robot as a task involving contact between the environment and a compliance tool.	
Experimental results show that the proposed method is superior in all evaluation items (Contact force error, Binary Accuracy, and Contact Line Error). Furthermore, by adding the data extension and wrench offset action proposed by the authors, the robot is able to scrape objects from the table while avoiding contact with obstacles.
strengths_and_weaknesses: The overall writing of the paper is easy to understandable, and the attached video also helped the reader understand. Most methods using supervised learning to learn robot policies from real-world data do not consider the adequacy of the generated trajectory or contact because they give the model predictions directly as motor commands. In contrast, the authors use MMPI to account for losses to trajectory and contact predictions, and they clearly describe their specific method. Furthermore, the authors also demonstrate the effectiveness of the proposed method by conducting multiple quantitative evaluations.

One concern is that there are few comparisons with other methods. Since the authors evaluated the accuracy of the task only based on differences in the structure of their proposed model (with or without vision and data augmentation, etc..), it is better to include the results of comparisons with other studies if the authors want to claim the effectiveness of the proposed method. Many previous studies of tasks involving contact exist. [1] showed that by using imitation learning to learn the contact force with the floor surface, the robot can properly mop the floor even if the grasping position and length of the mop are different. The reviewer has some major comments. The reviewer agrees with the issue on line 21. However, as the solution shown by the proposed method is limited, it would be better to show the consideration and results of the tool-environment interaction when the grasping position of the spatula is changed. Line 49 states that a variety of trajectories are realized, but basically the robot is just moving the spatula straight. Therefore, the diversity of trajectories is not shown. In order to demonstrate the potential of the proposed method, it would be helpful to have results for varying the positions of objects and obstacles. The reviewer understood that by collecting training data with a random action policy, contact information between the tool and the spatula could be collected. As shown in Fig. 2, the reviewers also understood that the robot predicts its behavior every step based on the observed data. On the other hand, obstacles and objects are not included in the training data. How do they recognize the objects and rub them with the spatula? It is unclear how the approach trajectory to the object is learned from the random trajectory. Explaining the data flow during learning and during execution will help the reader understand. Finally, how are the Desired contact trajectories in Figure 4 calculated? Are they arbitrarily determined by human?	
	
[1] Sakaino, Sho. "Bilateral control-based imitation learning for velocity-controlled robot." 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE). IEEE, 2021.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The lack of comparison with other previous researches makes it difficult to understand the effectiveness of the proposed method. The authors do not clearly state why their method is better for the issues mentioned in the introduction. In other words, the reviewer believes that the issue can be solved by other methods. It would be good to revisit additional tasks to show that the proposed method is effective in solving the issues.
issues: As mentioned in "Strengths And Weaknesses," the lack of comparative experiments with other research methods is critical. Additional experimental results are needed to show that the proposed method is effective in solving the problem.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper propose a method to learn the dynamics of these contact features from real world data with unknow tool geometry, and propose a controller that uses learned dynamics model for visuo-tactile contact servoing and show that it is effective at performing scraping tasks with a spatula.
strengths_and_weaknesses: Strengths:
1. propose a contact configuration: a binary contact mode; 
2. present a framework for modeling compliant tool-environment contact interactions by learning contact feature dynamics.
3. propose a learned model architecture to capture the dynamics of contact features, trained in a supervised fashion using real world self-supervised data.
4. design and demonstrate a controller using the contact feature dynamics to realize diverse goal trajectories.

Weaknesses:
1. authors believe the line contacts model can be straightforward to extend to patch contacts by using a richer contact descriptor, but do not proof it.
2. The model is used for compliant tool-environment contact interactions, but only test on a spatula.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the idea presented in this paper is interesting and promising.  The network architecture and data collection are clear.
More experiments are needed to proof the generalization of the framework or defined strict constraints.
issues: 1. Format of formulas (1), (2), (3)  and (4), (5) are wrong. The formulas (4),(5) are only one, not two.
2. For line contacts model, different geometry and physical properties (e.g., mass, stiffness, and friction) tools are needed to proof the generalization of the framework.
3. How about the performance on discontinuous surface, such as v-shape surface?
4. How about the performance of rotating the spatula comparing to pushing?
review_title: null

Meta-review: metareview: This paper presents an architecture for modeling the interaction of a compliant tool and the environment. The contact dynamics is represented by the contact indicator, contact line, and end-effector wrench. The method is demonstrated on hardware using a compliant spatula to scrape an obstacle.

The authors successfully addressed many of the reviewers concerns, most notably by adding more examples with multiple and unseen tools to demonstrate the generalization capability. The paper makes a valuable contribution to the conference by providing a novel solution to an interesting problem.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)