Learning the Dynamics of Compliant Tool-Environment Interaction for Visuo-Tactile Contact Servoing
Authors: Mark Van der Merwe
Dmitry Berenson
Nima Fazeli
Venue: CoRL 2022 Poster (2022)
Keywords: Contact-Rich Manipulation
Multi-Modal Dynamics Learning

Abstract: Many manipulation tasks require the robot to control the contact between a grasped compliant tool and the environment, e.g. scraping a frying pan with a spatula. However, modeling tool-environment interaction is difficult, especially when the tool is compliant, and the robot cannot be expected to have the full geometry and physical properties (e.g., mass, stiffness, and friction) of all the tools it must use. We propose a framework that learns to predict the effects of a robot's actions on the contact between the tool and the environment given visuo-tactile perception. Key to our framework is a novel contact feature representation that consists of a binary contact value, the line of contact, and an end-effector wrench. We propose a method to learn the dynamics of these contact features from real world data that does not require predicting the geometry of the compliant tool. We then propose a controller that uses this dynamics model for visuo-tactile contact servoing and show that it is effective at performing scraping tasks with a spatula, even in scenarios where precise contact needs to be made to avoid obstacles.

Learning the Dynamics of Compliant Tool-Environment Interaction for Visuo-Tactile Contact Servoing
Mark Van der Merwe
Dmitry Berenson
Nima Fazeli
Department of Robotics University of Michigan
(markvdm, dmitryb, nfz @umich.
Abstract: Many manipulation tasks require the robot to control the contact be- tween grasped compliant tool and the environment, e.g. scraping frying pan with spatula. However, modeling tool-environment interaction is difficult, es- pecially when the tool is compliant, and the robot cannot be expected have the full geometry and physical properties (e.g., mass, stiffness, and friction) of all the tools must use. We propose a framework that learns to predict the ef- fects of a robot's 's actions on the contact between the tool and the environment given visuo-tactile perception. Key our framework representation that consists of binary contact value, the line of contact, and an end-effector wrench. We propose method learn the dynamics of these contact features from real world data that does not require predicting the geometry of the compliant tool. We then propose a controller that uses this dynamies model for visuo-tactile contact servoing and show that it is effective at performing scraping tasks with spatula, even in scenarios where precise contact needs be made avoid obstacles.
novel contact feature
Keywords: Contact-Ricl Manipulation, Multi-Modal Dynamics Learning
Introduction
M11
111
SbT
End
Figure 1: We present method for extrinsic contact servoing, i.e., controlling contact between compliant tool and the environment. Our method able to complete the requested contact trajectory, avoiding contact with surface obstacles, and successfully scrape the target object. Note that to do this the spatula must be tilted so that only corner of is in contact.
Many manipulation tasks require the robot to control the contact between grasped tool and the environment. The ability to reason over and control this extrinsic contact is crucial to enabling helpful robots that can scrape a frying pan with a spatula, crase or wipe surface [1], screw bottle cap onto a bottle [2], perform peg-in-hole assemblies [3, 4], and perform many other tasks.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

In this work, we seek to address the problem of controlling the extrinsic contact between a grasped compliant tool (e.g. a spatula) and the environment. In general, the robot cannot expect have the full geometry and physical properties (e.g., mass, friction, stiffness) of all the tools it must use or the geometries of the environments it must manipulate in. Instead, the robot must utilize multimodal sensory observations, such as pointclouds and tactile feedback, to act on the environment.
In recent years, learning-based methods have become increasingly popular to address the complex- ities of robotic manipulation, including for contact-rich tasks [5]. These methods can be loosely grouped into model-free methods. that directly learn policy [3. 2, 6], and model-based methods, that learn system dynamics [7, 8, 9]. By focusing on modeling system dynamics, model- -baee meth- ods can plan reach new goals without retraining, and are often more data-efficient [9]. Therefore, we propose learning the dynamics of our system to solve the extrinsic contact servoing task.
It is not obvious which representation to use for these dynamics. Fully recovering tool and environ- ment geometries from visual data [10, 11] and tactile feedback [12] has been widely explored, with recent extensions to compliant geometries [13]: however, even if the system can be fully identified, contact models to resolve interactions can have limited fidelity [14]. On the other hand, learned dy- namics representations can be difficult to interpret and require demonstrations or observations from the desired state to specify goals [7, 15]. Instead, we propose a novel contact feature representa- tion for our learning method that focuses on tovl-enviromment interaction and bypasses explicitly modeling the whole system. We represent the contact configuration as 1) a binary contact mode (indicating if the system in contact); 2) a contact geometry (as line in 3D space); and 3) an end-effector wrench.
We propose learning architecture model the dynamics of the proposed contact representation from raw sensory observations over candidate action trajectories. We propose structuring the model as a latent space dynamics model with a decoder that recovers the contact state. We also propose an action offset term in the dynamics that allows us to accurately propagate robot poses, despite controller errors (e.g. from robot impedance). To provide labels to our model, we collect self- supervised data on a 7DoF Franka Emika Panda, using sensor data to automatically label contact state.
We validate our proposed method by completing various desired contact trajectories on the real robot system. We first show that our method can track diverse desired contact trajectories in the absence of obstacles. Next, we demonstrate that we can utilize extrinsic contact servoing to scrape a target object from the table, while handling occlusions and avoiding contact with obstacles (Fig. 1).
2 Related Work
Existing research has investigated the task of recovering contact locations. Manuelli et al. [16] local- ize point contacts rigid robot with known geometry by employing particle filtering approach to update a set of candidate contact locations based on force torque sensing. Kim et al. [4] and Ma et al. [17] model contact between grasped rigid object and the environment by assuming stationary line contacts and modeling the deformation of a GelSlim gripper. The estimated line contact is then used in a Reinforcement Learning (RL) policy. Neither of these methods extends to compliant tools and neither models the dynamics of the contact configuration.
Other works explore tactile servoing methods, where contact at the sensor is driven to desired configuration. Li et al. [18] use a large tactile pad and define contact configuration features of objects pressed against the sensor. They manually construct a feedback controller based on these features and use drive contacts to desired configurations. Sutanto et al. [19] use smaller profile tactile sensor and learn the dynamics of a learned latent space. They then employ a Model Predictive Control (MPC) scheme to drive contacts to desired configurations on the sensor. Both of these works assume contact is happening at the sensing location. We, on the other hand, seek to servo extrinsic contacts, where we do not get direct sensing at the point of contact.
Other work focuses on maintaining contact between tool and the environment. Sakaino [20] uses imitation learning to learn a controller able to maintain contact between mop and a tabletop. In contrast, we wish to not only maintain contact but control the extrinsic contact geometry.

---

3 Problem Formulation
We parameterize our contact feature as binary contact indicator c² E 10,1), used to indicate whether the tool is in contact, a contact line & E R2x3 representing the contact geometry between the tool and the environment, and an end effector wrench e E R The geometry e' is only active when the tool is in contact c² = 1. The contact representation allows extrinsic contact goals to be expressed as desired contact trajectories G 9g.g9.. 92........ ..... gL), where each gi E Ixx is desired contact line to reach. We assume that contact should be maintained throughout the task.
We formulate extrinsic contact servoing model predictive planning problem, given observations of the current state of the system Ou- For a given horizon T, we select the next 7 desired contact lines ".+1,-..,,i++77 C G to be our current contact goal sequence. The planning problem is:
T min Σ d(cr gete) ap:T-1
1=1
(1)
s.t. 1,
(c) CGTT CoT) g d0:T-1)
Here g is model describing the contact feature dynamics. The binary constraint ensures that the tool remains in contact while the cost function measures the distance between the two contact lines, the average Fuclidean distance between the line endpoints. Finally, if d(c), 9i+1) < increment i, thus moving to the next sequence of desired contact lines for the next round of planning.
Method
4.1
Contact Feature Dynamics Model
To solve our constrained optimization Eq. 1, we require a model which can map from raw obser- vations Op and proposed action trajectory a0:T-1 to the resulting contact states {cieTs co.r ). We propose modeling the contact feature dynamics as a deep neural network. Our actions are changes in end effector pose.
We assume access to a pointcloud vo and input wrench hg measured at the robot's wrist our observations, On = (vn, ltn). Note that end effector wrench is both an input to our method and part of the contact state; predicting future wrench aids the representation learning and provides expected wrenches for planning.
We perform all learning in the local end effector frame. We transform the pointeloud the end effector frame D and clip to 0.5m² bounding box region around the end effector that con- tains the contact event. We similarly predict our contact lines in the current end effector frame, E., ct, Vt E [1. Learning in the end effector frame provides invariance in the visual domain to translations and rotations of the end effector and removes distractors that do not contribute to the contact state, such as the rest of the robot arm or the scene background.
Our contact feature dynamics model (Figure 2) has three components: encoder which maps from raw observations to a learned latent space, a decoder d which maps from the latent space to the contact state, and dynamics model ff which captures dynamics in the latent space. We parameterize the models by a set of learned weights 0.
We start by embedding the current observations into the latent space with our encoder å0 e(vo hn). We unroll actions in the latent space as the contact state alone has insufficient con- textual information (e.g. end-effector pose and local geometry information) to predict the next contact state. Because we predict the tth contact state in the current end effector frame EE,, an important consideration when designing this frame. Controller error, e.g., from the impedance the robot, means the commanded action is not perfectly executed. To account for this, we predict additional term from our dynamics model Au which is an (3) transformation that predicts the offset between the commanded and realized next end effector pose. Thus, dynamics model predicts, 2,+1 A f(2,, a.). This allows us to construct the following recursive estimate of our end effector frame WT WīEE, (T (Aa441)) where WÊEE, is the SE((3) transformation describing the pose of the end effector at time t. We know the initial transform W "TE, from our robot proprioception, T(a:) pro-
dynamics model is being able to accurately recover

---

4
b
90
EE
00
Aa
d c
BE
K
d
vo
00
20
f
2
hp
Legend
Enozderr
Dysersis
Decoder
Figure 2: Our proposed contact feature dynamics model. Our architecture embeds raw observations into a latent space where dynamics can unrolled. We then decode the contact state from the latent space. We also predict action offset term in order accurately predict future robot poses.
vides the transformation for the action command, and T( T(A@++1 for the predicted offset term. To enforce valid (E(() predictions we predict rotations the axis-angle representation [21].
Finally, we recover our contact state estimates with our decoder d given the latent state 2,: &, EE c, 6t d(2z). We can then recover the predicted contact line in the world frame by com- posing with the estimate the end effector frame transformation W "TEE,- An overview of the model architectures is shown in Fig. 2 and full architecture details can be found in Appendix A.
4.1.1 Training Loss
We train our model on rollouts of the system, where a single example is a sequence [Uz, he, a,, Da cu ct c We define the loss over the example as:
Co BCF(C) +a. MSF(C, MSF(C", ,(r)
T
+ MSE(AA, Aa.) +7 MSE(2,,
Here BCE is the Binary Cross Entropy classification loss and MSE the Mean Square Error re- gression loss. Oz, [3, and y loss weighting terms. The first four loss terms are prediction losses over the contact mode, contact geometry, end effector wrench, and action offset transformation. The final loss term latent consistency loss, which encourages latent rollouts to match the latent state yielded by encoding future observations.
4.2 Extrinsic Contact Servoing Controller
We propose to solve planning problem using Model Predictive Path Integral (MPP)), which has been shown to effective for continuous control tasks where sampling is cheap and parallelizable (e.g., neural network representations) [22]. We convert binary constraint to penalty, penalizing a trajectory if it yields actions that lead out of contact. Pairing this with the contact line prediction
loss yields the following final cost function:
Σ dew W ai g.+)) +
if <
O.W.
The constraint is violated if the likelihood of binary contact below the classification threshold th, in which case penalize by the distance to the threshold. ő weights the penalty against the contact line loss. With this cost function we apply MPPI to yield the next action and execute on the robot.
4.3 Extrinsic Contact Dynamics Labeling
Our contact dynamics training loss in Eq. 2 requires ground truth contact state labels (C/, ct. ci") time l. As accurate simulation contactful interactions challenging, we propose method of data acquisition directly in the real world. To generate contact line labels, we use high resolution, low frequency scanner, Photoneo PhoXi 3D Scanner, to generate high quality seans of the contact interaction. Using these scans, generate contact line labels by filtering points just above the table,

---

Contact (4)
Contact Force
Dut Mocel
Vosu Only
3,50
No Otfict
0:85-
3eary
0.75
(250
0.75
6:3
0.70
003
S0mp
strp
Stop
(b)
(c)
Figure 3: Contact Feature Dynamics Performance: results show the importance of modeling the action offsets and the compliance of the tool; without both, contact line estimates drift. The Rigid Body Baseline does not predict contact wrenches, so is not shown in (c).
clipped to the area around the end effector. We then cluster these points to remove noisy points on the tabletop and generate the contact line cj by selecting the two furthest points in the cluster. See Appendix B for examples of contact labels. We use force torque sensor identify the contact state wrench ct We identify binary contact &' automatically from whether line was found in the pointcloud and/or based on force torque sensing.
5 Results
5.1
Experimental Setup
We test our method on a Franka Emika Panda with rigidly compliant spatulas at the end effector (Fig. 1). For our observations On, sensor' and mount an ATT Gamma Force/Torque sensor between the end effector and tool. We use the last four wrench values received after the previous action completed as the tactile input hg. To collect our datasets, we use a random action policy with a heuristic to encourage contact between the tool and the spatula. No other objects are on the tabletop during data collection to allow proper data supervision, as detailed in Sec. 4.3.
pointclouds Vn from Intel Realsense D435
5.2 Baseline
We compare our proposed method to modeling the contact dynamics as rigid system. We assume the commanded actions are perfectly executed by the robot to recover the future poses of the end effector. We assume access to the tool geometry as pointcloud in the end effector frame. This pointcloud is then transformed via the future poses of the end effector to recover where the tool would be, assuming rigid motions. We further assume that we know the table location and identify any points in the transformed point cloud that penetrate the table surface. If any exists, set e == 1. We then choose the two furthest points in the intersecting set of points as the end points of the contact line e'_ The baseline does not predict the wrench e but is enough solve our planning problem in Eq. 1. This baseline makes three assumptions our method does not make: 1) it assumes access to a pointcloud of each tool, it assumes knowledge of the current tool being used, 3) assumes explicit knowledge of the environment.
5.3 Modeling Contact Feature Dynamics
We first investigate the ability of our model to capture the contact feature dynamics exhibited in our dataset. We train three variations on our model. First is the full model, as described in Sec. 4.1, hereafter called "Full Model." Second, to understand the importance of modeling the action offset of the robot, we ablate the offset action prediction, thus we propagate the end effector frame only with the commanded action. We call this method "No Offset." Finally, we investigate our model trained only on visual input data, called "Vision-Only."
We train a dataset collected from three spatulas (see "Training Tools" in Fig. 5a), collecting 200 trajectories on each, for total of 30000 transitions. We split the data 80/10/10 for train, validation, and test. We train with rollout horizon of T 3. All methods are trained with the Adam opti-
We don't 't use the high-fidelin fidelity Photoneo scan is very low- fow-feeueency scanner.
5

---

020
-00
030
40
4.30
438
400
433
650
(a) Straight
(b) V-Shape
(c) Tilt
(d) Tilt Back
Figure 4: Qualitative Scraped Areas: our "Full Model" controller (blue) is able to closely match desired contact trajectories (red).
mizer [23] until convergence on the validation set. We a 100.0, 3 = - 0.1 in our loss term in Eq. 2 to balance the scale of the terms.
We compare the prediction performance of the models the test split of the dataset in Fig. (see Appendix C for torque prediction results, whose results are very similar to force results). Our full model achieves high accuracy in predicting binary contact (> 95%), 3-5mm contact line error, and less than 0.5N force error. Comparing "Full Model" to "No Offset" shows the importance of modeling action offsets, without which contact line error quickly grows. Comparing all learned models to the Rigid Body Baseline (Sec. 5.2) shows the importance of modeling the compliance of the tool. The "Vision-Only" model unsurprisingly struggles to recover contact forces. Appendix show additional results examining generalization of the model to the dynamics of an unseen spatula.
5.4 Extrinsic Contact Servoing
Next, we investigate how our proposed controller performs following specified contact trajectories.
Obstacle-Free: We start by attempting to servo along four different contact trajectories in the obstacle-free environment. The desired contact trajectories are shown in Fig. 4, and explore transla- tion of contact well as cases where the robot must tilt the tool to achieve contact smaller than the width of the tool. We use the same labeling technique introduced in Sec. 4.3 to get ground truth contact trajectories executed by the controller. We the experiment once on training spatula (left-most in Fig. 5a) and once on an unseen spatula (right-most in Fig. 5a).
We use the controller described in Sec. 4.2, with &i = 0.45, 0.05 and compare our "Full Model" learned dynamics (as trained in Sec. 5.3) vs the "Rigid Body Baseline" dynamics. To investigate the planning performance, we run the controller five times per trajectory and measure the Intersection over Union (IoU) of the desired and swept contact areas. We construct the goal contact area by sweeping the space between the specified goal contact lines and the realized controller swept area by assuming that the space between two consecutive contact states was swept out if the two states were both in contact.
We show qualitative examples of the "Full Model" controller realized scrapes on the training spatula compared to the goal scrapes in Fig. 4. We see that the controller is able to closely match the desired swept areas, including in the difficult tilting problems.
The loU performance shown for the training spatula (Fig. 5b) and unseen spatula (Fig. 5c). Our proposed method outperforms the baseline nearly all cases, for both the training and unseen tool runs. Performance drops for all methods on the tilting problems, as it is more difficult to maintain dexterous contact only a part of the tool.
We next examine our method's 's robustness to visual occlusions and reaction forces arising from contact with target object to be scraped. This task is common in construction, cooking, and cleaning. A deformable and slightly adhesive material (Playdough) is pressed onto the surface and a contact trajectory is specified through the object. In one case, the target object is alone on the tabletop (Fig. 6), and thus we specify a contact trajectory using the full width of the tool. In the second scenario, obstacles contact trajectory that avoids them. All experiments are performed on the leftmost spatula in Fig. 5a. We our "Full Model", and train 22005 sequences collected only with the relevant tool.
With Obstacles:
on the table near the target object (Fig. 1), thus must specify
Besides running our Full Model in these scenarios, we investigated enhancements of the method to aid its performance in the presence of visual occlusions and object reaction forces. First,

---

Training
kp is the Unseen Spatula
111
Frisl Budyy Bood Inc
Tot Hode:
0.8
Starp Arna Wel
0.0
Usace
0.0
V-Scapa Topeewyy
Srright
Decc
Content Traeetory
(8)
(b)
(C)
Figure 5: (a) Tools used for experiments. (b),(c) Extrinsic contact servoing IoU performance training spatula (b) and unseen spatula (c). Our proposed method tracks the desired contacts with higher IoU, compared to rigid body baseline method, even when running on an unseen tool.
620
111
ShNr
Frd
Figure 6: Example of extrinsic contact servoing execution for the "Straight" target obstacle scrape experiment. Our method is able to accurately servo along the desired contact trajectory and success- fully scrape the target.
applicd data augmentation to our training dataset, randomly generating ellipsoids in the pointcloud and using hidden point removal algorithm [24] to provide corresponding occlusions in the original point cloud. See Appendix for examples augmented inputs. We call this method "Full Model Aug." Second, we investigate using the difference between the predicted and observed wrenches Aw द." c," to derive an action offset to compensate for the extra wrench experienced by the robot. From Aw we derive an action that will counteract this wrench offset à = Aw pose gain of the impedance controller. The offset action is composed with the original action from the controller. We call this method "Full Model + Aug + Wrench Offset."
he
1.50
1.0
0.95
0.
0.85
0.8
Nodel
0.50
Full Model Aug
9.4
PJll Model Aug e wannh uffiser action
0.70
02.
0.55
gh
(a) Percent Mass Removed
Percent Footprint Removed
Figure 7: Target Scraping Results: Our "Full Model" and variations for addressing visual occlusions and reaction forces arising from contact with the target object perform comparably on both metrics over 5 trials on each experiment.

---

We use two metrics. First, we measure the approximate mass of the target object to be scraped before and after scraping and determine the percentage mass successfully removed. Second, we compare the 2D) footprint of the material before and after scraping and report the percentage of the footprint successfully removed. The second is more challenging metric, since even a slightly wrong scrape will leave residue. The quantitative results over 5 runs of each method in each experiment setup are shown in Fig. 7. Examples of scrape executions are shown Fig. and Fig. See Appendix C for more examples of scrape results.
We see that in each case, all methods were able to remove over 95% of material mass on average and about 40-60% of material's footprint from the tabletop. Surprisingly, we don't I see a consistent improvement training our model with visual occlusions adding action offsets. The Full Model's robustness to occlusions here could be due to the fact that we use single tool in these experiments, and thus it may sufficient most cases to capture the location of the table with respect to the end effector in order to estimate the contact line. Even with visual occlusions near the tool contact, is likely our method can still recover the relative pose of the tabletop from the surrounding points. The lack of clear improvement from the wrench offset action may be due to the fact that it is sufficient to be able to replan, as we do at every step with our MPPI controller.
6 Limitations and Conclusion
Limitations: A common failure mode for our method is in controlling contact when the tool is tilted, where it is more likely for the method to yield actions that take the robot out of contact. This could, in part, be due to data imbalance. In future work, we are interested in utilizing online learning [9] or curiosity [25] to more effectively cover the space of contacts our dataset.
There are cases where tool to environment contact is not represented well as a contact line. Extend- ing our contact feature dynamics these tasks will require expanding our representation learning method to consider these more diverse contact specifications and more complex contact modes.
Finally, method relies upon supervision. For future contact-rich tasks of interest, the need for labels could become more costly. We hope to investigate how we can remove reliance upon supervi- sion by exploring few/zero shot generalization [26, 27] and domain randomization techniques [28].
Our approach simplifies contact rich interactions for compliant tool manipulation, by avoiding the necessity for full system identification while maintaining interpretability and accuracy by explicitly modeling the contact state of the system and how evolves. In the future, we wish to investigate method's applicability contact state is crucial, such as wiping with a cloth. Additionally, we wish to investigate representa- tions that handle more complex contact geometries.
Conclusion:
other tasks where full state estimation is difficult, but the
8

---

Acknowledgments
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 1841052. Any opinions, findings, conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work was supported in part by Toyota Research Institute under the University Research program 2.0. This work was supported in part by ONR grant N00014-21-1-22 --21-1-211111 18 and NSF grants IIS-1750489 and IIS-2113401.
References
[1] R. Martín-Martin, M. A. Lee, R. Gardner, S. Savarese, J. Bobg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pages 1010-1017. IEEE, 2019.
[2] Levine, C. Finn, Darrell, and P. Abbeel. End-to- training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1)13334-1773, 2016.
[3] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg. and J. Bohg. Making sense vision and touch: Self-supervised learning multimodal representations for contact-rich tasks. In 2019 International Conference Robotics and Automation (ICRA), pages
IEEE, 2019.
[4] S. Kim and A. Rodriguez. Active extrinsic contact sensing: Application to general peg-in- hole insertion. In 2022 International Conference Robotics and Automation (ICRA), pages 10241-10247, 2022. N001100119//CRA4463699 2022446699922229812211111
[5] O. Kroemer, S. Niekum, and G. D. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of machine learning research, 22(30), 2021.
[6] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4- 37(4-5):421-436 2018.
[7] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pages 564-574. PMLR, 2021.
[8] P. Mitrano, D. McConachie, and D. Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics, 6(54):eabd811 170, 2021.
[9] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference ceedings of Machine Learning Research, pages 2555-2555 2565. PMLR, 09-15 Jun 2019. URL https::/proceediing mlr, ppwss/v97
Machine Learning, volume 97 of Pro-
http//999/haaaffffee /taa/aafffee.. html.
[10] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[11] C. B. Choy, D. Xu, Gwak, K. Chen, and Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pages 628 644. Springer, 2016.
[12] D. Watkins- Valls, Varley, and Allen. Multi-modal geometric learning for grasping and manipulation. In 2019 International conference on robotics and automation (ICRA), pages 7345. IEEE, 2019.
[13] Y. Wi, Florence, A. Zeng, and Fazeli, Virdo: Visio-tactile implicit representations of de- formable objects. IEEE International Conference Robotics and Automation (ICRA), 2022.

---

[14] N. Fazeli, S. Zapolsky, E. Drumwright, and A. Rodriguez. Fundamental limitations in perfor- mance and interpretability of common planar higid-body contact models. In Robotics Research, pages 555-571. Springer, 2020.
[15] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised cor- respondence in model- reinforeement learning. In Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 693-710. PMLR, 16-18 Nov 2021. URL https: /ptpa/////ccceiigas mlr, ppwsss/115/mmanullllll. html.
[16] L. Manuelli and R. Tedrake. Localizing external contact using proprioceptive sensors: The contact particle filter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5062-5069. IEEE, 2016.
[17] D. Ma. S. Dong, and A. Rodriguez. Extrinsic contact sensing with relative-motion tracking from distributed tactile measurements. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11222-1126. IEEE, 2021.
[18] Q Li, C. Schürmann, R. Haschke, and H. Ritter. A control framework for tactile servoing. In Robotics: Science and systems. Citeseer, 2013.
[19] G. Sutanto, N. Ratliff, B. Sundaralingam, Y. Chebotar, Z. Su, A. Handa, and D. Fox. Learning latent space dynamics for tactile servoing. In 2019 International Conference on Robotics and Automation (ICRA), pages (622--3622 IEEE, 2019.
[20] S. Sakaino. Bilateral control-based imitation learning for celocity-controlled robot. In 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), pages 1-6. IEEE, 2021.
[21] A. Byravan and D. Fox. Se3-nets: Learning rigid body motion using deep neural networks.
In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 173-180, 2017. doi: 10.1 109/CCRA.0017.99999
[22] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE Inter- national Conference Robotics and Automation (ICRA), pages 1714-1721, 2017. doi: 10.109/CRAA2201..7992000
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htLp///://aree http:://aawvvv..aaaaeeeeee wwtp:o//////....////./
[24] S. Katz, A. Tal, and R. Basri. Direct visibility of point sets. ACM SIGGRAPH 2007 papers, pages 24-es. es. 2007.
[25] S. Rajeswar, C. Ibrahim, N. Surya, F. Golemo, D. Vazquez, A. Courville, and O. Pinheiro. Haptics-based curiosity for sparse-reward tasks. In Conference on Robot Learning, pages 395- 405. PMLR, 2022.
[26] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pages 4367- 4375, 2018.
[27] F. Zhao, J. Zhao, S. Yan, and J. Feng. Dynamic conditional networks for few-shot learning. In Proceedings of the European conference on computer vision (ECCV), pages 19-35, 2018.
[28] Mitrano and D. Berenson. Data augmentation for manipulation. Robotics Science and Systems (RSS), 2022.
[29] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
10


Reviews:
[{"review_id": "yKJPZmOt0JW", "reviewer": "Reviewer_r4to", "summary": "This paper presents a model-based contact servoing framework to perform control the contact between a compliant tool and the robot's environmet. Contact is parameterized as a binary contact flag, line of contact and an end-effector wrench. Dynamics are learned in a latent space with an encoder-decoder framework that is used to predict the contact parameters at every step using a pointcloud observation and the input wrench measured at the robot's wrist.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Data collection procedure is self-supervised without needing human intervention or labelling.\n- In-depth analysis of different ablations of the model to study and evaluate the effectiveness of different components of the system\n\nWeaknesses:\n- Generalization: Experiments are restricted to a single spatula. Furthermore, the surface in contact with the spatula also remains the same. Would be interesting to see generalization to new spatulas, or even just train/test on more tools as well as analyzing variations in other parameters such as the contact surface.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While it would be interesting to see the performance of the proposed framework in noisier, more realistic settings, the method is promising and the presented analysis is sound and well-executed. The paper is able to demonstrate controlled extrinsic dexterity with a deformable spatula while using an impedance controller, which is a promising direction as we start moving towards robots operating in human environments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- It would be nice to see experiments and analysis of generalization, both in terms of the generalization of the model to other tools, environments, surfaces etc as well as the generalization of the method to more general, parameterizable contact shapes.\n- One baseline that would be interesting to compare against, especially in the cases of tracing contact areas where the tool is NOT tilted, would be to apply a larger force than required and using just vision to track the line. While this would not be able to handle partial contacts, it would be interesting to see if this method can solve the non-partial contact servoing tasks (like the straight area) as well as the proposed method."}}, {"review_id": "nmprgRSkpwO", "reviewer": "Reviewer_DDzt", "summary": "The paper propose a method to learn the dynamics of these contact features from real world data with unknow tool geometry, and propose a controller that uses learned dynamics model for visuo-tactile contact servoing and show that it is effective at performing scraping tasks with a spatula.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. propose a contact configuration: a binary contact mode; \n2. present a framework for modeling compliant tool-environment contact interactions by learning contact feature dynamics.\n3. propose a learned model architecture to capture the dynamics of contact features, trained in a supervised fashion using real world self-supervised data.\n4. design and demonstrate a controller using the contact feature dynamics to realize diverse goal trajectories.\n\nWeaknesses:\n1. authors believe the line contacts model can be straightforward to extend to patch contacts by using a richer contact descriptor, but do not proof it.\n2. The model is used for compliant tool-environment contact interactions, but only test on a spatula.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the idea presented in this paper is interesting and promising.  The network architecture and data collection are clear.\nMore experiments are needed to proof the generalization of the framework or defined strict constraints.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Format of formulas (1), (2), (3)  and (4), (5) are wrong. The formulas (4),(5) are only one, not two.\n2. For line contacts model, different geometry and physical properties (e.g., mass, stiffness, and friction) tools are needed to proof the generalization of the framework.\n3. How about the performance on discontinuous surface, such as v-shape surface?\n4. How about the performance of rotating the spatula comparing to pushing?"}}, {"review_id": "nfygY63FDbtR", "reviewer": "Reviewer_yQUC", "summary": "The paper presents a robot control framework for controlling contact forces at the tip of a pre-grasped tool. Remarkably, the proposed approach considers a situation where the compliance of the grasped tool can be used to change the geometry of the contact. The authors presents a method to learn a dynamic model which allows to predict the effects to the robot actions on the contact geometry and on the robot end-effector location. This dynamic model is used in an MPC framework to control contact forces in a variety of scenarios, including controlling the contact forces at the tip of a flexible spatula to scrape a target object in presence of obstacles.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a novel solution to a novel problem. Controlling tool contact forces is a very interesting problem which (to my knowledge) has been addressed only in the rigid case. The submitted paper considers the non-rigid case which is a significant novelty. The major limitation of the paper is the need of a complicated data-gathering phase which requires precision equipment (Photoneo) and human supervision. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The submitted paper presents a novel task and an interesting solution. The paper is well written and easy to follow. Videos are interesting to watch and the proposed solution is sound and technically correct.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- ***Action specifications***. At line 93 author mention the action trajectory $\\mathbf{a}_{0:T-1}$ but it isn't clear what is the action space used for the robot. Authors should make this clear specifying if they control torques, positions and/or impedance.\n\n- ***Typo on line 88***. $g_{i}+1$ should be $g_{i+1}$.\n\n- ***Typo on line 81***. $\\mathbb{R}^{2,3}$ should be $\\mathbb{R}^{2 \\times 3}$.\n\n- ***Improve notation***. Since $G = \\[ g_1, \\dots, g_i, \\dots, g_L\\]$ are trajectories in time I suggest the notation $G = \\[ g_1, \\dots, g_t, \\dots, g_T\\]$."}}, {"review_id": "haHoaiFXKG", "reviewer": "Reviewer_gMWE", "summary": "The authors proposed a learning approach for modeling tool-environment interaction which learn the dynamics from real world data.\t\nKey of proposed method is embedding the latent spaces from the robot's sensor data and decodes the contact feature representation which consists of binary contact value, the line of contact, and an end-effector wrench.\tThe authors verified the scraping operation of a clay-like object using a spatula attached to the tip of an actual robot as a task involving contact between the environment and a compliance tool.\t\nExperimental results show that the proposed method is superior in all evaluation items (Contact force error, Binary Accuracy, and Contact Line Error). Furthermore, by adding the data extension and wrench offset action proposed by the authors, the robot is able to scrape objects from the table while avoiding contact with obstacles.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall writing of the paper is easy to understandable, and the attached video also helped the reader understand. Most methods using supervised learning to learn robot policies from real-world data do not consider the adequacy of the generated trajectory or contact because they give the model predictions directly as motor commands. In contrast, the authors use MMPI to account for losses to trajectory and contact predictions, and they clearly describe their specific method. Furthermore, the authors also demonstrate the effectiveness of the proposed method by conducting multiple quantitative evaluations.\n\nOne concern is that there are few comparisons with other methods. Since the authors evaluated the accuracy of the task only based on differences in the structure of their proposed model (with or without vision and data augmentation, etc..), it is better to include the results of comparisons with other studies if the authors want to claim the effectiveness of the proposed method. Many previous studies of tasks involving contact exist. [1] showed that by using imitation learning to learn the contact force with the floor surface, the robot can properly mop the floor even if the grasping position and length of the mop are different. The reviewer has some major comments. The reviewer agrees with the issue on line 21. However, as the solution shown by the proposed method is limited, it would be better to show the consideration and results of the tool-environment interaction when the grasping position of the spatula is changed. Line 49 states that a variety of trajectories are realized, but basically the robot is just moving the spatula straight. Therefore, the diversity of trajectories is not shown. In order to demonstrate the potential of the proposed method, it would be helpful to have results for varying the positions of objects and obstacles. The reviewer understood that by collecting training data with a random action policy, contact information between the tool and the spatula could be collected. As shown in Fig. 2, the reviewers also understood that the robot predicts its behavior every step based on the observed data. On the other hand, obstacles and objects are not included in the training data. How do they recognize the objects and rub them with the spatula? It is unclear how the approach trajectory to the object is learned from the random trajectory. Explaining the data flow during learning and during execution will help the reader understand. Finally, how are the Desired contact trajectories in Figure 4 calculated? Are they arbitrarily determined by human?\t\n\t\n[1] Sakaino, Sho. \"Bilateral control-based imitation learning for velocity-controlled robot.\" 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE). IEEE, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The lack of comparison with other previous researches makes it difficult to understand the effectiveness of the proposed method. The authors do not clearly state why their method is better for the issues mentioned in the introduction. In other words, the reviewer believes that the issue can be solved by other methods. It would be good to revisit additional tasks to show that the proposed method is effective in solving the issues.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "As mentioned in \"Strengths And Weaknesses,\" the lack of comparative experiments with other research methods is critical. Additional experimental results are needed to show that the proposed method is effective in solving the problem."}}]

Author rebuttal:

Decision: Accept (poster)