Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing
Authors: Leon Kim
Yunshuang Li
Michael Posa
Dinesh Jayaraman
Venue: CoRL 2023 Poster (2023)
Keywords: contact perception
manipulation
vision-based

Abstract: Contacts play a critical role in most manipulation tasks. Robots today mainly use proximal touch/force sensors to sense contacts, but the information they provide must be calibrated and is inherently local, with practical applications relying either on extensive surface coverage or restrictive assumptions to resolve ambiguities. We propose a vision-based extrinsic contact localization task: with only a single RGB-D camera view of a robot workspace, identify when and where an object held by the robot contacts the rest of the environment. We show that careful task-attuned design is critical for a neural network trained in simulation to discover solutions that transfer well to a real robot. Our final approach im2contact demonstrates the promise of versatile general-purpose contact perception from vision alone, performing well for localizing various contact types (point, line, or planar; sticking, sliding, or rolling; single or multiple), and even under occlusions in its camera view. Video results can be found at: https://sites.google.com/view/im2contact/home

Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing
Leon Kim, Yunshuang Li, Michael Posa, and Dinesh Jayaraman
GRASP Laboratory
University of Pennsylvania, USA {leonmkim, sheylali, posa, dineshj]@seas upenn edu
Abstract: Contacts play a critical role in most manipulation tasks. Robots to- day mainly use proximal touch/force sensors to sense contacts, but the informa- tion they provide must be calibrated and inherently local, with practical ap- plications relying either on extensive surface coverage or restrictive assumptions resolve ambiguities. We propose vision-based extrinsic contact localization task: with only single RGB-D camera view of robot workspace, identify when and where an object held by the robot contacts the rest of the environ- ment. We show that careful task-attuned design is critical for a neural network trained in simulation to discover solutions that transfer well to a real robot. Our fi- nal approach im2contact demonstrates the promise of versatile general-purpose contact pereeption from vision alone, performing well for localizing various con- tact types (point, line, or planar; sticking, sliding, or rolling; single or multiple), and even under occlusions in its camera view. Video results can be found at: attps://siten. www.oe.. www./vview/////ooaact..t..
Keywords: contact perception, manipulation, vision-based
Introduction
Perceiving and reacting to contact is critical for performing manipulation tasks [1-5]. Consider what happens when person puts book on a crowded shelf: they hold the book and aim for gap until meets resistance, jostle the book to make room, then press sideways to line up the book against its neighbor, and finally slide the book snugly into place. Throughout, the key events they must track all have to do with the physical contacts between various surfaces: when and where they are made, broken, and transition from sticking sliding. What means does a robot have today sense such contacts between its body, its tools, and external objects?
Current contact perception techniques for robots operate mainly from force torque and touch sens- ing. Force torque (F/T) sensors [6] located at the robot's joints can inform model-based contact estimation techniques [7-11]. However, this estimation problem under-determined. To see this, consider two-fingered manipulator in two contact configurations: applying a 10 N force one finger, versus applying a 5 N force each finger. A wrist F/T sensor located directly behind the gripper's midpoint is aliased: it senses identical forces in both cases. For such direct contacts with the robot, aliasing can be partially resolved with proximal touch sensors [12-15] applied to con- tacting surfaces on the robot, but over only a limited area. Further, consider as in our introductory example, a robot holding book that contacts a bookshelf. Such "extrinsic" contact still only sensed indirectly at the fingers, and remains impossible to resolve. To overcome these issues, to- day's contact estimation techniques reduce the number of unknowns by operating under restrictive assumptions, such as about the number and types of contacts, and the shapes of the various objects involved (Sec 2). Finally, even when operating within these assumptions, F/T and touch sensors often expensive, and drift or deteriorate quickly over use [15-18] requiring frequent and cumber- some re-calibration.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Prediction
Probahility
>
Figure I: Sequence of predictions of our method im2contact for localizing extrinsic contact in an image between grasped object (such spatula) and the environment (such as a bowl)
To address both the lack of glohal information available to F/T and touch sensors, and the sensor drift issue, we instead operate from camera images. Cameras can see the entire scene, and function without deterioration for long durations. Since contacts function of the shapes and movements of the objects and the environment, contact estimation would be easy if we could first get perfect 3D shape and pose estimates from vision. However, we argue that contact from 3D would be a poor solution. First, given occlusions around the critical contact regions and arbitrary-shaped objects, near- wear-perfect shape and pose estimates would require impractically many camera views. Next, the often-crucial binary distinction between near-contacts and true contacts can be hard capture by continuous-valun object pose estimates.
Rather than infer contacts from intermediate visual representations such as shapes and poses, we propose im2contact, more direct approach for data-driven, model-free visual contact location estimation at the outputs of a deep neural network. Our system is trained entirely in simulation and incorporates the combination of cropped depth images of the salient regions of the scene, an additional reference depth image to specify the grasped object, and motion cues from optical flow.
In our zero-shot transfer evaluations real robot, im2contact models demonstrate the possibility purpose contact perception from vision alone, performing well for
and promise of versatile general-purpose localizing various contact types (point, line, or planar; sticking, or sliding; single or multiple), and even under occlusions in its camera view.
2 Related Work
The robotics community has long recognized the crucial role of external contact sensing manipu- lation, particularly of contacts between the manipulator and environment (e.g. s. [8, 9, 19]), coined by Bicchi et al. [7] as "intrinsic contact". However, "extrinsic contact," e.g. between a held object and the environment, is similarly useful, but notably more difficult. Prior work extrinsic contact has primairily focused on the use of force and touch sensors and typically found success through strong assumptions which ultimately limit the potential scope of the results. For example, related work has relied on the assumption and/or enforcement of pre-defined contact configurations [20-22], limiting application to grasped objects seen during training [23], or tight coupling with information gather- ing motions [24]. Other approaches have incorporated pointclouds with force and touch sensing, but assume full coverage of grasped objects [25, 26] or restrict interactions to line contacts [27].
In order to enable robots to readily use unmodeled tools in unstructured environments, many of the above restrictions must be lifted. We make steps towards this via the choice of vision-based sensing with no explicit assumptions made on possible contact configurations, object properties, and minimal access to privileged knowledge of the object or environment.
3 Visually Localizing Extrinsic Contacts In General Manipulation Scenes
For robot arm holding grasped object, interested in localizing contact between the held object and the rest of the environment, as the robot arm moves in its workspace. Unlike much of the

---

a
(b)
(c)
(d
(e)
Prediction
Probubility
Figure 2: U-Net Depth generally performs well in sim but poorly on real data challenging cases such as: (a,d) occluded contact (b,e) ambiguous grasped object geometry (c,f) near-contact with background object
prior work described above, we do not assume access to prior information about the held object or the environment, other than the table.
As inputs at each time instant, we assume a HxW W view of the robot workspace from single fixed RGB-D camera as shown in Fig 3, alongside proprioceptive sensing of the robot state from joint encoders. This is a minimal sensing setup for vision-base based robot manipulation, chosen to maximize the scope of our problem formulation.
As output, would like system to generate H W map of estimated contact locations, that can be overlaid on the camera image, similar to dense image-based features [28, 29]. To achieve this, we will train neural networks with standard pixel-wise binary cross-entree cross-entropy classification objectives. This treats the the output contact map values at each pixel as contact probabilities, and maximizes the likelihood under the model of the annotated ground truth contact locations in the training data.
3.1 Simulated Training Data
is not practical to obtain ground truth contact location annotations from real video, but fortunately simulators provide this information. To generate target contact maps for training, we project 3D contact points from the Gazebo simulator [30] into the camera frame. Next, we generate training data in episodic fashion. In each 15-second episode, we randomize the geometry, masses, fric- tion coefficients, and initial poses of grasped and environment objects. Shapes are chosen to be cylinders, spheres, and cuboids with random parameters. The grasped object is rigidly attached to the robot arm throughout the episode, as the robot end-effector moves to randomly set targets with a low-impedance controller to generate rich interaction data. Additional details are included in Appendix A. Our 4500 episodes (675000 frames at 10 fps) of training data span many types of extrinsic contacts: point, line, and plane contacts, instantancous collisions and sustained sliding or rolling contacts along with simultaneous contact with multiple bodies.
3.2 A Simple Baseline To Illustrate The Difficulty of Sim-To-Real Transfer
To motivate final method, we first showcase the difficulty of sim-to-real transfer with baseline, U-Net Depth. U-Net Depth builds on the widely used U-Net [31, 32] architecture for totageto--imae problems. In the U-Net, an encoder first assimilates information from over the entire 240x320 input image into a "bottleneck" representation of size x5x 20x 1024. Then, a decoder iteratively spatially upsamples this representation with the aid of skip connections from intermediate encoder layers, to finally produce an output over the original x40x320 input dimensions. facilitate sim-to-real transfer and access 3D information, U-Net Depth uses depth images [33] as input rather than color images. We train using pixel-wise cross entropy.
U-Net Depth trains to near-zero training losses, and performs very well on held-out data in simu- lation. However, on real data, its performance deteriorates significantly. In Fig 2, we anecdotally note few challenging scenarios where this baseline struggles in the real world: (a) under significant occlusion, it misses the occluded contact and predicts irrelevant false positives on the table, (b) when the geometry of the grasped object is ambiguous, predicts contact between an extraneous box and table, and (c) it produces far more false positives in near-contact scenarios than in sim. We more thoroughly evaluate U-Net Depth in Sec 4.
3

---

RGB image
Output contact map
depth
crop
optical
Depth image
flow
object
U-Net
reference
Figure 3: im2contact architecture. Depth and flow are both cropped and concatenated before being passed through the U-Net. The object reference is passed through separate encoder whose output is concatenated at the bottleneck of the U-Net. The final output of the U-Net is a probability map over the original image dimensions.
3.3 Facilitating Generalizable Contact Localization With im2contact
To mitigate the failures of U-Net Depth on real data, we consider the possible causes of poor gener- alization. Machine learning systems commonly over rely on spurious correlates of the target labels in the training data [34-36], which fails to generalize under distribution shift. During training, spu- rious correlates could distract the optimizer [36] from finding true "causes"; this can alleviated by anticipating such distracting but irrelevant correlates in advance and removing them from the learner's s inputs. Inputs might be also incomplete, lacking information the true underlying cause of the outputs. We propose three potential improvements U-Net Depth, one remove distracting correlates, and two to add missing causal information into the inputs.
very large, all extrinsic contacts will occur close to the end-effector. With calibrated camera and proprioceptive knowledge, is easy to locate the end-effector in the image. We propose to focus the network on the most relevant regions by cropping the depth images into a 90x 10 box around this end-effector location at each time, before feeding them to the U-Net. addition, we add three channels for coordinate convolution as proposed by Liu et al. [37] to provide the network with spatial relationship of the cropped image with respect the original image.
Depth Cropping (+ crop): Assuming grasped objects that
Grasped Object Reference (+ obj-ref): We have thus far reduced contact localization an instantaneous task, performed from the current sensory observations. However, consider the sce- nario when the grasped object is occluded by the environment: if we do not know its spatial extent, it may be impossible to know whether there is any extrinsic contact, Similarly, in cluttered views it may be difficult to disambiguate between the boundaries of the grasped object and the envi- ronment from a single depth image. We propose to provide such missing information through an additional input: a single reference depth image of the grasped object in the robot gripper, partially specifying the shape of the grasped object.
Optical Flow (+ flow): From single image, it may be difficult, even impossible, to differen- tiate between contact and near-contact. For example, gap between the grasped object and the environment of a few millimeters is likely imperceptible without a perfect vantage. However, con- tact forces often induce motion in the environment, causing environmental objects to slide or roll. To capture these cues, we propose to use optical flow computed from the camera RGB images.
In the rest of the paper, we use the name im2contact for the combined approach: U-Net Depth + crop + obj-ref + flow. Fig 3 schematically depicts the network with all inputs and outputs.
Reducing To Discrete Contact Locations. Our training procedure generates contact probability maps, but is convenient for evaluation and useful for downstream tasks identify discrete contact locations. To this end, adapt the greedy non-maximum suppression (NMS) technique [38-40], commonly used in object detection. To identify spatially separated peaks in the contact probability

---

map, we first reject all pixels scoring below e = 0.01, sort pixels by descending score, and iteratively reject pixels within 5 px neighborhood of higher-scoring pixels. See Appendix for pseudocode.
Implementation Details. The pixel-wise cross-entropy loss treats each pixel independently of all others and hence penalizes a px deviation in predicted location equivalently to 100 px deviation. To abate this, we spatially blur ground-trus -ruth contact maps before computing our training target, and find that this accelerates training. We monitor performance held-out simulation data to implement early termination. For computing optical flow, use the off-the-shelf RAFT [41] model. Code and models will be available at our website.
4 Experiments
We test im2contact in simulation and in real to evaluate performance in realistic tabletop manipu- lation settings. We further test, via ablation studies, whether, and under what settings, our proposed changes (crop, obj-ref, and flow) do improve real performance. We include additional anecdotal experiments which push im2contact beyond its training settings, evaluating generalization.
Real Robot Data, Annotations, and Performance Metrics. We perform teleoperated experi- ments in a table-top environment with a Franka Emika Panda robot arm and an Intel Realsense L515 RGB-D camera, with reasonably well-matched simulation and real robot setups, as shown earlier in Sec 3. To facilitate some quantitative comparison, we manually annotate contact locations for 30 episodes (approximately mins comprising 12.362 frames, of which 1/3 involve least one con- tact) of real robot interaction data, spanning variations in the grasped object, environment objects, robot movements, and contact scenarios. Recall that for simulation experiments, annotated contacts are readily available as in Sec 3; we use 500 episodes (75,000 frames) for computing simulation performance metrics.
Our main metrics are precision, recall, and their harmonic mean, the Fl score. All three must in [0, 1]. and higher is better. To compute them, we first follow the procedure from See 3.3 for reducing the network's ' output probability maps discrete contact locations. We then match each ground truth contact point to predicted contact point with the Hungarian algorithm, and drop predictions where the ground truth is over 15px away as false positives. For our metrics, ground truth contacts with matches are true positives, and those without are false negatives. We run 5 random seeds for each method and report means and standard errors.
Quantitative Results on Simulation and Real Data. Fig 4 plots contact localization precision and recall in real (hollow circles) and simulated (solid circles) test data, for im2contact, the base- line U-Net Depth (Sec 3.2), and leave-one-out ablations of im2contact that in turn drop crop, obj-ref, and flow. Tabular results are presented in Appendix C.
First, for the simple U-Net Depth baseline, these results clearly validate our qualitative observations from See 3: precision and recall both deteriorate dramatically from sim to real. Next, U-Net Depth and im2contact are both among the best performing methods on sim data, but im2contact only drops marginally in performance on real data, so it remains among the best-performing methods in real data, while U-Net Depth performs worst.
Moving on to the Icave-one-out ablations, "w/o crop" deteriorates nearly as much as U-Net Depth, while "w/o obj-ref" and "w/o flow" degrade more gracefully upon transfer to real data, crop con- tributes the most among the three components im2contact. Fig (middle) plots standard error for real data and the legend (right) lists F1 scores. From this initial coarse analysis with limited quantitative metries, losing obj-ref marginally hurts precision, and losing flow does not significantly affect these scores. Note that these aggregate scores might not reflect performance in rare scenarios, of which there are many.
5

---

Sim-To- Reaall
Performance on Real Data
0.55,
Simulation
METHOD: FI SCORE
Real
0.50
U-Net Depth: 0.38
0.50
Precision
Precision
0.45
w/o crop: 0.4000
0.45
w/o obj-ref: 0.52#0.006
w/o flow: 0.53
0.40
0.40
im2contact: 0.53+00005
0.35
0.35
0.4
0.5
0.6
0.4
0.5 Recall
Recall
Figure 4: (Lefft) Degradation under sim-to-real transfer (Middle) Precision-Recall real data (Right) Legend along with Fl scores on real data.
Qualitative Analysis. Armed with the initial evidence from the coarse quantitative results, we performed thorough qualitative analysis over the aforementioned 30 real robot episodes. Figs and present keyframes of selected video reels. Fig compares the outputs of all methods at some selected frames across the dataset. These examples are selected to illustrate some key insights from more comprehensive analysis (remaining examples available on website):
As our metrics above suggested, crop is indeed the most important contributor to im2contact performance. In Fig 7, both im2contact "w/o crop" and U-Net Depth perform similarly poorly, demonstrating the criticality of focusing the model on relevant regions for finding good solutions.
The grasped object reference frame (obj-ref) frequently improves performance in occluded contact situations (Fig 5, left), when the grasped object moves coherently with others, creating a mis- leading flow field (Fig 5, right and Fig 7 (d)). This is consistent with our arguments in Sec 3.3; in such situations, knowing the shape of the grasped object is critical for localizing extrinsic contacts.
Consistent with the metrics above, the effects of optical flow are less obvious, and w/o flow is very similar to im2contact in Fig 7. We have however anecdotally observed that improves
idx= 1 12
idx= 153 153
idx=328
idx=46 461
im2contact
wi abj-ref'
Probubility
Figure 5: im2contact w/o obj-ref: obj provides useful information about the grasped object's shape, which is not otherwise available under occlusions or misleading flow fields. Optical flow fields overlaid on all images. (Left) The grasped can behind the cereal box. slides up its side to topple the box. Here, the grasped object is occluded. (Right) The robot pushes two boxes to topple them, then pushes one of them back upright. These sustained pushes lead to coherent flow fields that do not help to separate grasped object from neighbor. In both cases, "w/o obj-ref" errs by treating the can-cum-boxee collective as single grasped object, generating false positives locations where the boxes contact their neighbors.

---

idx=485
idx=571
idx= 1 1046
idx= dix=105m 1052
im2contact
w/o flow
Obj-ref Frame
Annotation
Prediction
Probability 2-
Figure 6: im2contact w/o flow (Optical flow fields overlaid only on im2contact images): (Left) The robot uses a grasped can to press against and pivot a box into an upright position. The reference frame does not clearly show the can's shape, flow must separate the grasped object from others. Sure enough, in frame 485 during the motion, "w/o flow" predicts irrelevant contacts at the foot of the shelf, but im2contact performs correctly. Later, in frame 571 immediately after the motion, im2contact no longer has the flow cue, and fails. (Right) The robot uses a large box to sweep number of cans under shelf. In this highly cluttered scene, "w/o flow" consistently predicts false contact between a stationary can in the foreground rather than the true contact happening in the background underneath the shelf. im2contact the other hand the flow the moving background can to localize the true contacts.
performance in certain cases. In Fig 6 (a), flow suppresses false positive in near-contact situation and helps identify contact with can within clutter in Fig 6 (b).
huttof-Distribution Evaluation of im2contact We conclude with brief highlight reel in Fig 8 visualizing the output of im2contact on chosen out-of-distribution settings that are diffi- cult to annotate manually. Despite significant distribution shift from the training domain, we find im2contact still produces reasonable estimates in examples including moderately deformable ob- jects and human demonstrations. The promising results on the latter suggest our extrinsic contact estimates may be amenable for use in guiding robot policy learning from human videos, previously demonstrated by Bahl et al. [42] in the context of intrinsic contacts. The details of our human demonstration evaluations are included in Appendix E and additional examples are our website.
5 Limitations and Future Work
Our method, im2contact, using only single camera and proprioception, localize contacts in the 2D image. While this already surprising and useful, there is much information that system does not capture: it does not localize contacts in 3D, perceive contact forces, classify modes of contact. Our deliberately simple sensing setup might be fundamentally limited for solving these broader problems. For example, predictions over the RGB-D image may be backprojected into 3D) spatial coordinates using the depth map. However, single-viewporis depth images are only 2.5D representation of geometry, so we are unable localize full 3D coordinates of contact points during occluded contacts. Multiple cameras may help address this issue.
In addition, our cropping method assumes an upper bound on the dimensions of the grasped object. Integrating an off-the-shelf object segmentation method may enable more more adaptive schemes for cropping and grasped object specification that may also improve robustness to occlusions.
Lastly, our method leverages the global information provided by vision over more local tactile sens- ing. In doing so, sacrifice the precision of our method which is prone false positives during near-contact and occlusions, even with the addition of optical flow. Our preliminary efforts to inte-
7

---

imZcontact
wro abi-tef
w/o
wro crop
L-Net Depti
(a)
Contact clitter
Amb iubiguuus geometry
Occluded push push
(d)
Annotation
Prediction
Probability
Figure 7: Our full ablations on four illustrative samples (flow visualized on the models that use them): (a) grasped cereal box encounters numerous near-contacts in highly cluttered scene. (b) From the same episode, the grasped cereal box comes into contact with a foreground can in the cluttered scene. (c) A can stacked on top of another in a cluttered scene (d) A significantly occluded grasped sugar box pushes a can across the table
Deformable object
Human Demonstration
Prediction
Probubility E
Figure 8: im2contact predictions on (Left) interactions between two moderately deformable plushies (Right) human video of inserting book into tight space
grate F/T sensing in Appendix D validate that F/T sensing can improve performance in sim, but will require substantial effort to transfer to real which we hope to actualize in future work.
6 Conclusion
We present a method that enables extrinsic contact localization from vision alone with minimal prior knowledge or explicit assumptions on the grasped object and environment. By incorporating well-motivated inputs to our model in simulation, we show successful sim-to-real transfer of our model whereas a naive baseline fares poorly. The method also shows promise on our chosen out- of-distribution settings that include deformable objects and human demonstrations. In future work, hope to address the limitations of im2contact and explore its downstream utility in enabling control tasks involving tool assembly.

---

Acknowledgments
We would like to thank Denny Cao for his contributions to the synthetic data generation pipeline and teleoperation interface, well the reviewers for their insightful feedback during the rebuttal period. Leon Kim was supported by the NSF GRFP, Yunshuang Li by the Chiang Chen Overseas Fellowship, Michael Posa by NSF CAREER Award FRR-2238480, and Dinesh Jayaraman by NSF CAREER Award 2239301.
References
[1] R. Johansson and J. Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience, 10(5):345-359, 2009.
[2] R. D. Howe. Tactile sensing and control of robotic manipulation. Advanced Robotics, 8(3): 245-261, 1993.
[3] S. Tian, F. Ebert, D. Jayaraman, M. Mudigonda, C. Finn, R. Calandra, and S. Levine. Ma- nipulation by feel: Touch-based control with deep predictive models. In 2019 International Conference on Robotics and Automation (ICRA), pages 818 824. IEEE, 2019.
[4] A. Aydinoglu, Sieg, V. M. Preciado, and M. Posa. Stabilization complementarity systems contact-aware controllers. IEEE Transactions on Robotics, 38(3):1735 1754, 2021.
[5] M. Romano, Hsiao, G. Niemeyer, S. Chitta, and K. Kuchenbecker. Human-inspired robotic grasp control with tactile sensing. IEEE Transactions Robotics, 1766):0077- 2011.
[6] M. Y. Cao, S. Laws, and F. R. Baena. Six-axis force/torque sensors for robotics applications:
A review. IEEE Sensors Journal, 21(24)27238-27251, 2021.
[7] A. Bicchi, J. K. Salisbury, and D. L. Brock. Contact sensing from force measurements. 12(3): 249-262. ISSN 0278-3649. 10.1177700 7w11177/027886499300000000
URL https : h/tp://ooooopppe org/
Publisher: SAGE Publications Ltd STM.
[8] A. De Luca, A. Albu-Schaffer, S. Haddadin, and G. Hirzinger. Collision detection and safe reaction with the DLR-III lightweight manipulator arm. In 2006 IEEE/RSJ International Con- ference on Intelligent Robots and Systems, pages 1623-1630. ISSN: 2153-0866.
[9] L. Manuelli and R. Tedrake. Localizing external contact using proprioceptive sensors: The contact particle filter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5062-5069. IEEE. ISBN 111--11-5900377622 doi:10.1 (09/ROS.2016. 7759743. URL htttt::://eeeeplorrr. ieee. orgg//ocument/7555443/// ottpr///esoo.ooooooooooott/////
[10] A. Zwiener, C. Geckeler, and A. Zell Contact Point Localization for Articulated Manipulators with Proprioceptive Sensors and Machine Learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 323-329. 100:10.1100/1C1000.... 109./..A00/// 2.46666 URL https: http://eeeexepeeeeeee ieee.
[11] T. Pang. J. Umenberger, and R. Tedrake. Identifying external contacts from joint torque mea- surements on serial robotic arms and its limitations. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6476-6482. doi: (0L.1009//.AA44502222220000090 9966766... ISSN: 2577-087X.
[12] A. Fishel and G. E. Loeb. Sensing tactile microvibrations with the ciotac-comparison with human sensitivity. In 2012 4th IEEE RAS & EMBS international conference biomedical robotics and biomechatronics (BioRob), pages 1122-1127. IEEE, 2012.

---

[13] W. Yuan, S. Dong, and E. H. Adelson, Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[14] M. Lambeta, P.-W. Chou, S. Tian, Yang, Maloon, R. Most, Stroud, R. Santos, A. Byagowi, G. Kammerer, al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-338 3845, 2020.
[15] R. Bhirangi, T. Hellebrekers, C. Majidi and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. arXiv preprint arXiv:21 2021.
[16] B. Wu, Wu, and Shen. Research on calibration system error of 6-axis force/torque sensor integrated in humanoid robot foot. In 2010 8th World Congress Intelligent Control and Automation, pages 6878-6882. IEEE, 2010.
[17] F. J. A. Chavez, G. Nava, S. Traversaro, F. Nori, and D. Pucci. Model based in situ cali- bration with temperature compensation of 6 axis force torque sensors. In 20/9 International Conference on Robotics and Automation (ICRA), pages 5397-5403. IEEE, 2019.
[18] M. Reinvee, K. Jansen, al. Utilisation of tactile sensors in ergonomic assessment of hand- handle interface: review. Agronomy Research, 12(3):907-914, 2014.
[19] P. Grady, J. A. Collins, S. Brahmbhatt, C. D. Twigg, C. Tang, J. Hays, and C. C. Kemp- Visual pressure estimation and control for soft robotic grippers. 2022 IEEE/RSI In- ternational Conference on Intellieett Robots and Systems (IROS), pages 0628-3635. doi: 10.1009/RR0S4711.22222000
ISSN: 2153-0866.
[20] K.-T. Yu and A. Rodriguez. Realtime state estimation with tactile and visual sensing for in- serting suction-held object. 1oi:10.1 109//ROS.201.855000 88594877
[21] D. Ma. S. Dong, and A. Rodriguez. Extrinsic contact sensing with relative-motion tracking from distributed tactile measurements. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11262-1 1268. doi: 1011111199/RA488000 2021 99561781. ISSN: 2577-087X.
[22] S. Kim, D. K. Jha, D. Romeres, P. Patre, and A. Rodriguez. Simultaneous tactile estimation and control of extrinsic contact. URL o http://xxvi.orgrbb////aae http//b.b..33303355... 03385.
[23] A. Molchanov, O. Kroemer, Z. Su, and G. S. Sukhatme. Contact localization on grasped objects using tactile sensing. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 216-222 doi: 1017/000.....11.755555..
2016.7759058.
ISSN: 2153-0866.
[24] Y. Karayiannidis, C. Smith, F. E. Vina, and D. Kragic. Online contact point estimation for uncalibrated tool use. In 2014 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 2488-244 2494. IEEE. ISBN /atc///mmtt//00220//// 6907206. URL http: /tt:://e/eeeeeeeeeeee ieee. org/
doi: 109/CCAA.2014
[25] C. Higuera, S. Dong, Boots, and M. Mukadam. Neural contact fields: Tracking extrinsic contact with tactile sensing. URL http://arxiia wwtp://////aa//ae//eeea 09297.
[26] A. Sipos and N. Fazeli. Simultaneous contact location and object pose estimation using pro- prioception and tactile feedback. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3233-3240. doi: 100334444..011111110//0/////011 10.11099/RR034471222022299999999 2153-0866.
ISSN:
[27] M. Van der Merwe, D. Berenson, and N. Fazeli. Learning the dynamics of compliant tool- environment interaction for visuo-tactile contact servoing. URL http: attp://arxits org/at org/abs/ abs/ 2210. 03836.
10

---

[28] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, N. Fazeli, Alet, N. C. Dafle, R. Holladay, I. Morona, P. Q. Nair, D. Green, I. Taylor, W. Liu, T. Funkhouser, and A. Rodriguez. Robotic pick-and-place with multi-affordance grasping and cross-domain image matching. URL http://arxiv. org/abs/ /0tb.///11.... 01330.
novel objects in clutter
[29] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. URL http: htattt:://riivv www. 08756.
[30] N. Koenig and A. Howard. Design and use paradigms for gazebo, open-source multi-robot simulator. In 2004 IEEE/RSI International Conference on Intelligent Robots and Systems (IRES)(EEEE Cat. No. 04CH37566), volume 3, pages 2149-2154. IEEE, 2004.
[3]] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Ditevention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[32] N. Siddique, Paheding, C. P. Elkin, and V. Devabhaktuni. U-net and its variants for medical image segmentation: A review of theory and applications. leee Access, 1-0051,,,7, 2021.
[33] M. Danielczuk, M. Matl, Gupta, A. Li, A. Lee, J. Mahler, and K. Goldberg. Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data. In 2019 International Conference on Robotics and Automation (ICRA), pages 7283-7290. IEEE, 2019.
[34] H. Ye, C. Xie, T. Cai, R. Li, Z. Li, and L. Wang. Towards a theoretical framework of out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:23519- 23531. 2021.
[35] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wich- mann. Shortcut learning in deep neural networks. Nature Machine Intelligence, ((11):666--6 2020.
[36] V. Nagarajan, A. Andreassen, and B. Neyshabur. Understanding the failure modes of out-of- distribution generalization. arXiv preprint arXiv:2010.15775. 2020.
[37] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018.
[38] G. Burel and D. Carel. Detection and localization of faces on digital images. Pattern Recogni- fion Letters, (5(10)9933966) 1994.
[39] P. Felzenszwalb, D. McAllester, and D. Ramanan. A discriminatively trained, multiscale, deformable part model. In 2008 IEEE conference on computer vision and pattern recognition, pages 1-8. Icee, 2008
[40] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
[41] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. URL http:
// //arxiv. org/
12039.
[42] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as versatile representation for robotics. URL http::/ttt/ttiii org/ aa/aaaba/2300 /2044. 08488.

---

A Randomized Simulation Data Generation
Environment objects: At the beginning of cach 15 second episode, 8 objects are spawned into the environment with either box or cylinder geometry with equal probability. We do not include spherical geometries as they are rare in household/kitchen settings, though we do introduce them into the possible grasped object geometries.
The spawn position of each object is sampled uniformly in a box above the table (z: [0.2m, 0.8m], [-0.38m, 0.38m], 2: [0.1m, 0.4m]). Euler angles are sampled uniformly between Σх. The respective dimensions for each primitive geometry (height, width, length for box, diameter and length for cylinder, and diameter for sphere) is sampled uniformly between 0.02m and 0.3m, mass sampled uniformly between 0.05kg and 2kg, and friction sampled uniformly between 0 and
The grasped object is spawned with the same procedure used to generate the geometry, dimensions, mass, friction above, but the position is sampled uniformly in cylinder defined in the end-effector frame with radius 0.001m and end-points at m0.007m and 0.001m along the z-axis. The orientation is sampled uniformly in a cone about the z-axis of the end-effector frame with an aperture angle of 0.7x.
Grasped object:
Robot policy: Desired delta end-effector positions are sampled at 50Hz from an Ornstein- Uhlenbeck (OU) process which is tracked by impedance controller. Desired orientation is con- stant and chosen to match the world frame, though we use very low orientation stiffness in the impedance controller to encourage diverse orientations during contact.
In the general form of the OU process: de, 0(u x4)) dl + a dW,, the first term deterministic and draws the process back to constant y (referred to "drift") with linear gain 0, while the second term is the stochastic wiener process where the variance is scaled by
The desired x, y trajectory is sampled independently with different parameters from the desired trajectory as we'd like to keep the motion in the xy-plane as diverse as possible, while ensuring the end-effector is close enough to the table to make frequent contact with the environment objects and table surface.
For the z, y process, we define four halfspaces in polar coordinates that contribute to the first drift term in the OU process to keep the desired trajectories within polar rectangle around the robot's s workspace. These terms only become active when the end-effector leaves the respective halfspace. Hence, when the robot is within all halfspaces, the x, process becomes simply a wiener process. The polar rectangle boundaries are defined such that the radius between 0.35m and 7m and the angle between 2.2 radians. For f we choose both x, y to be 20 and for the variance matrix choose diagonal with both elements equal to 0.22
For the = process, we choose a drift that is 0.2m above the table surface, 0 equal to 1, and a variance of 0.052
B Adapted Greedy NMS Algorithm
Algorithm Adapted NMS Algorithm
Input: PE [0, 11 HxW CCN' N2
Pe E [0, 1], Pams E RR,, Kiter E
Output:
1 - ((i.g) Paj > pt}
2 C C--orrt Cort(C,by by P.jj. order = descending)
3 for k: ko 1, mnin(Ktter CC) do
C+Cy ((i,)) [|[(ii iz, -jn)3)11 Time and Pas < Panixe 5 end for
6 return C
12

---

C
Tabular Results
Recall f
Precision 1
F1 t
Avg. TP distance +
method
U-Net depth
0.577 0.012
0.524 = 0.014
0.548 + 0.003
3.317 = 0.052
w/o obj-reffflow
0.585 + 0.011
0.467 = 0.008
0.519 - 0.004
3.42 = 0.08
w/o obj-ref
0.359 + 0.01
+ 0.002
0.528 + 0.005
3.341 + 0.048
w/o flow
0.574 + 0.011
0.518 + 0.01 0.01
0.544 + 0.004
3,453 + 0.029
w/o crop
0.551 0.011
0.53 = 0.015
0.539 0.005
3.265 = 0.061
im2contact
0.613 0.017
0.514 = 0.017
0.558 + 0.005
3.526 + 0.05
Table I: Metrics on simulation data with all ablations
Recall f
Precision 1
F1 f
Avg. TP distance +
method
U-Net Depth
0.41 = 0.022
0.36 = 0.017
0.38 + 0.01
4.1 + 0.138
w/o obj-ref+flow
0.58 0.018
0.44 + 0.014
0.5 + 0.009
4.09 + 0.109
w/o obj-ref
0.57 0.009
0.47 + 0.006
0.52 + 0.006
4.07 + 0.103
w/o flow
0.56 0.019
0.5 + 0.011
0.53 + 0.005
3.92 + 0.085
w/o crop
0.44 0.025
0.39 + 0.025
0,41 = 0.007
4.47 = 0.065
im2contact
0.56 = 0.022
0.5 + 0.017
0.53 + 0.005
4.08 = 0.094
Table 2: Metries on real data with all ablations
D Preliminary Evaluations of Adding Force-Torque Sensing
We have found in our initial efforts integrate force-torque sensing that this does indeed improve performance in simulation, with increase in F1 score from 0.56 to 0.60 sim-to-real transfer is challenging because of calibration errors, drift, and deterioration. To account for this, we compensate the available joint-torque sensing with model of the robot to- gether with estimated grasped-object mass to approximate the external joint-torques which can be attributed to contact. However, unmodeled effects remain: robot joint friction is difficult to identify well, and object inertial properties are only coarsely estimated. As a result, im2contact + F/T per- forms worse in real data: Fl score drops from 0.53 0.50 (Table 4). We show qualitative video examples on real data comparing im2contact with the addition of F/T sensing at our website: https://sites: google www//wwwwwiwaiaaacccccceeaeae.ee wtt.eeeeimmmcctt....mmmmm
in Table 3. How-
We may conclude that F/T sensing does not offer a simple silver bullet solution for consistently improving extrinsic contact sensing in our settings. Incorporating additional sensors to improve the performance of our vision-only im2contact approach will require non-trivial additional contribu- tions.
Recall f
Precision 1
F1
Avg. TP distance |
method
im2contact+F/T
0.62 1 0.004
0.584 + 0.011
0.601 + 0.006
3.507 + 0,042
im2contact
0.613 = 0.017
0.515 0.018
0.558 0.005
3.526 0.05
Table 3: Metrics on simulation data comparing im2contact with and without F/T sensing
13

---

Recall f
Precision ?
F1
Avg. TP distance
method
im2contact+F/T
0.479 + 0.024
0.54 + 0.017
0.504 + 0.011
4.069 + 0.123
im2contact
0.56 + 0.022
0.501 0.017
0.526 0.004
4.078 0.093
Table 4: Metrics on real data comparing im2contact with and without E/T sensing
D.1 Implementation Details of Adding Force- Prree-Torque Sensing
The Franka Panda robot provides estimated external joint-torques by compensating the joint-torque measurements with internal model of the robot's inertial and kinematic properties. These are then transformed into "virtual" external force-torque measurement at the end-effector by applying the pseudo-inverse of the jacobian, followed by alignment into the world-frame.
We attempt compensate the unknown grasped-object's inertial effects by coarsely estimating the induced gravitational wrench. On real data, assume the robot does not move the beginning of the episode and average the first 0.5 seconds of the world-frame z-component of the external wrench, followed by division by gravitational acceleration to obtain an estimated mass. In sim, we simply use the ground-truth object mass. In both sim and real, We assume the CoM location is fixed and located 15m along -Z-axis of the end-effector frame to obtain the adjoint map to compute the approximate gravitational wrench as a function of the end-effector pose.
On real data, apply causal low-pass Butterworth filter to reduce the observed force-torque oscillations during free-space motion that we suspect are attributable to coupled effects between the unmodelled joint-friction and the impedance controller.
We integrate the current-most external wrench estimate to im2contact by first passing the 6- dimensional wrench vector through small MLP, followed by tiling and concatenation the 15 x 1024 bottleneck of the U-Net. We train with the same training procedure, hyperparam- eters, and dataset before.
E Implementation Details of Human Demonstration Evaluations
We localize the human hand in the image by affixing passive reflective ball to the hand which can easily be thresholded and localized in the RGB-D camera's 's infrared image stream. We apply the same 90x 10 cropping window to the ball pixel coordinate with relative shift downward by 55 pixels.
We modify the cropping window hyperparameter during training of im2contact by additionally shifting the window down by 14 pixels to mitigate effects of domain shift in the agent's end-effector. Otherwise, we retain the same training procedure, hyperparameters, and dataset as before.
14


Reviews:
[{"review_id": "hrkGqgNXaV", "reviewer": "Reviewer_25ZM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors aim to sense contacts between a robot and a tool with the environment using a single rgbd camera. They use UNet Depth and cropped images of the robot's tool and optical flow as some of the inputs into their network. To get the training data for their network, they use Gazebo to get ground truth contacts with a variety of primitive objects. They show that some of the baselines and their method are able to have sim 2 real transfer with some hand labeled contacts. In their rebuttal, they added new experiments using a human hand to interact with objects and predicting the contacts as well as using force torque data to reduce the amount of false positives.", "summary_of_recommendation": "I have increased my decision from a weak reject to a weak accept due to the new experiments and real world results. While I still believe that this approach of using only a singular rgbd camera to detect contacts between a tool and the environment is not very practical, I think that the idea is still not that bad and could be at least published. As long as the authors can incorporate the results from the rebuttal nicely into the paper, I could argue that it is better than a weak reject but still worse than a strong accept. I still think that in reality multiple cameras could be used to get a nice mesh reconstruction of the environment which would aid in better contact detection \"without touch or force sensing\". In addition, currently the sim 2 real transfer only works if the camera and robot are only in the same configuration. If the authors could leverage NERF to be view agnostic, it would be great.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. I agree with your limitation that in not using the robot's proprioceptive feedback such as forces and torques, you could reduce the number of false positives I saw in your supplementary videos. It would be great if you could give some initial results of fusing the robot's force and torque data to improve the real world predictions. \n2. I also agree with your limitation that a lot of false positives and contact localization could be improved with more cameras. My question is why you had to put the realsense camera in front of the robot. From your motivation in the introduction, you mentioned wanting to put a book in a bookcase which requires a lot of contact sensing. However, it is unrealistic that in a real domestic mobile robotic setting that you would have a camera always in front of the robot. So how would your method work from around a head mounted view like from the Nimbro avatar robot? (https://www.youtube.com/watch?v=8AwgGSpcAe8)\n3. What I find most promising for this work is if it could be applied to human egocentric videos like epic kitchens and ego4d where humans do not have force torque sensors unlike the Franka robot. What do you think are the limitations to making your pipeline work on those videos?\n4. Also you should put a table in the main paper and not just on the supplementary website. I think it's easier to read a table than Figure 4.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have no problem with the quality or clarity of the writing or presentation. I just have issues with the motivation and the use cases of this proposed method of detecting contacts from a single view that is directly in front of the robot. In addition, there are many false positives still because only one camera is used. I do think the use of simulation is good for collecting ground truth contact data, but the data seems to have only been collected from one view. My main review is contained in the questions for rebuttal and the summary of recommendation. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "bVexzYQ2FzQ", "reviewer": "Reviewer_rgiT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a vision-based method for estimating contact localization without force sensing based on the motivation that robots today do not have any approach to reliably sense contacts in general settings due to the limitations in sensing technologies or sensor drifts. The proposed method relies only on a single RGB-D camera view of the robot's workspace and learns to identify when and where contact exists. Using simulated data for training, the authors proposed a method that transfers well to an actual robot with data, demos, and results presented in this paper and on a public website.", "summary_of_recommendation": "I do not recommend this paper due to the fundamental limitations of the proposed method and the limitations presented in this work. It could be helpful when the scenarios are simple enough or highly structured, which can be alternatively solved using the proposed method for automation purposes or cost consideration. But if any complexity is involved, the method would fail badly due to a lack of local contact sensing with physics-based training data and scenarios, which is fairly limited in applications. \n\nUpdated recommendation:\nAfter reviewing the rebuttal made by the authors, I would like to update my recommendation to weak accept. The paper is technical alright, and the idea is interesting in general. I would request the authors to revise statements such as \"yet robots today do not have any approach to reliably sense contacts in general settings\" and similar others. Also, the authors should strengthen the motivation of this paper so that it reflects a closer connection to robotics.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. \"robots today do not have any approach to reliably sense contacts in general settings.\" This statement is a bit absolute and harsh. Can the authors clarify what you mean by general settings? How reliable do you think that is necessary? Is it actually the case that we \"do not have any approach\" for sensing contacts? \n\n2. The authors claim that the proposed method shows a promising application for \"versatile general-purpose contact perception from vision alone\" ... \"and even under occlusions in its camera view.\" Can the method estimate contacts when a few balls are placed inside the bowel while conducting experiments similar to those in Fig. 1?\n\n3. Can the authors provide a more realistic application of the proposed method with a more detailed description that it can perform superior that using existing methods or force sensing technology? And can the authors also demonstrate scenarios where the proposed method may fail but the existing force sensing technology still works?\n\n4. Can the authors comment on whether we should move towards a direction where vision-base method only should be used for contact estimation or if the force sensors could still be of any use in the future of contact-related robot learning research and applications.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses an interesting topic, can a learning algorithm \"guess\" where and when contact happens, similar to humans watching a video and predicting if contact occurs based on prior knowledge or experience of environmental physics? The general idea is fine based on the close resemblance to human prediction and the results presented. It seems that based on the authors' claim that \"yet robots today do not have any approach to reliably sense contacts in general settings\" and the method presented in this work, this paper is proposing a solution that contacts estimation from a purely vision-based approach is what this paper is trying to promote for contact-based robot learning as a future direction. I think the statement itself is wrong (we have force sensors commercially available on robots, tactile arrays, or e-skins capable of distributed contact sensing, etc., which is abundantly available in the current literature). The problem is probably related to a disentanglement between the development of contact sensors and how they can be integrated with robotic solutions in applications instead of completely disapproving of this research direction. \n\nThe fundamental logic behind the proposed method still relies on the underlying physics of contact for collecting training data, either from simulations built based on such logic or from the reality that the authors gathered from the FT sensors from the robot's wrist. The vision-based observation serves only as an external view easily influenced by the lack of local contact or anomalies from data trained with \"common\" scenarios. For example, what if there are N number of objects placed inside a bowel, can the proposed method predict contacts using external images such as those in Fig. 1 to make the prediction using a spatula controlled by a robot while moving around? The limitation of this proposed method is obvious by nature, and the claim made by the authors is way too broad and affirmative which is just surprising. \n\nIt seems Sim-2-Real transfer is what the authors are really trying to demonstrate as a superior performance of this proposed method, which I think is mostly based on simplified task scenarios and structured settings that the model overfits for overconfident results. As I mentioned, what if placing a few objects inside the bowel or something behind the object with image occlusion and will the method still work? I think this method is fundamentally limited in application scenarios that do not fit a generalizable contact localization close to reality, where things like occlusion happen (almost) all the time. \n\nAnd based on the results presented, I don't see a clear classification or recognization of contact type identification for point, line, or planar; sticking, sliding, or rolling; single or multiple. In other words, I think the results presented in this work can be resolved by adding or choosing a suitable force sensing technology at the fingertip or object surface and identifying using physics-based analysis of the contact nicely. \n\nThe images taken for this paper are hard to see with the limited resolution.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "_h2JP-s-QL0", "reviewer": "Reviewer_qKK8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the problem of vision-only contact prediction. The authors propose to use a variant of a U-Net to predict pixelwise probabilities of a contact between an object held by a robot and the environment. To supervise this network, they use contact point labels generated in a simulator (Gazebo) and modify the architecture slightly (providing context views, cropping the image to only include the object, and providing optical flow data) to improve performance. While training on simulated contact data raises a significant sim-to-real issue, the authors study their method's performance on real objects and observe relatively good performance.", "summary_of_recommendation": "While I think the results of the paper are sound and correct, I have pretty serious concerns about the motivation -- i.e., it's unclear why predicting pixel-wise contact probabilities is useful for downstream robotics tasks. Unfortunately, many of the contributions of this work are closely connected to the pixel-wise representation and don't necessarily generalize to other representations, such as Signed Distance Functions (SDFs). \n\nHowever, with either a firm rebuttal from the authors or good discussion from other reviewers, I am open to having my mind changed.\n\nPost-rebuttal edit: My mind was changed. I think the authors can/will address my concerns, and found the additional human experiments to be a very good example of how the representation is useful/generalizable. While I personally am skeptical about the longer-term usefulness of the result, I think it's an important direction to explore and the results are surprisingly good for the simple setup.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Can you discuss more the practical usefulness of a pixel-wise contact representation? What manipulation problems do you think this contact/shape representation would be useful for? Is it easy to generalize to multiple objects?\n\nDo the ablation studies in Fig. 4 show that cropping is really the only added feature that improves performance significantly? Why include the additional information/complexity required (e.g., optical flow) if the performance gains are relatively small?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I found this paper to be somewhat mixed in its efficacy. In general the methods the authors use seem well-posed, and supervising contact location prediction using simulation seems like a reasonable approach. Further while the lack of baselines (since the authors have posed a relatively novel problem) makes it difficult to judge the overall performance of the final model, he contact probability heat maps it returns for novel objects, e.g., Figure 8, seem reasonable, if somewhat messy. \n\nHowever, since the authors are studying a relatively new problem, I think it is important for them to make the case that this problem is of interest to the broader robotics community. In my opinion, the paper would be greatly improved by a more thorough discussion of the authors' overall motivation -- how does pixel-wise contact prediction connect to canonical manipulation tasks like tool use, object pile manipulation, assembly, etc? \n\nThis justification is especially important since other works studying contact prediction, e.g., ContactNets [1] (uncited here), use signed distance representations, rather than predicting contacts in pixel space, which is a more intuitive/useful representation for performing things like in-hand manipulation, etc. I think the paper would also improve from a discussion of the benefits of a pixel-wise contact representation (which has many ambiguities, e.g., if a line contact lies along the camera's view direction) compared to other representations.\n\nThese motivational concerns aside, however, I found the paper to be clearly written and the approach sound. \n\n[1] ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representations, Pfrommer et al., CoRL '20.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "FAjDgdxXbx", "reviewer": "Reviewer_PPMJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "im2contact is an approach to estimate extrinsic contacts (i.e., contact between an object held by a manipulator with the environment) by looking at an RGB-D third-person view of the robot and its workspace, and additional cues such as optical flow and reference images of the grasped object. The approach is trained in simulation and deployed on a real-world robot manipulator (a tabletop workspace).", "summary_of_recommendation": "The paper presents an interesting method for extrinsic contact localization from RGB-D images. The proposed approach is sound and interesting; the paper looks well-executed. There are a few (minor) clarifications I would like to see discussed over the author response and revision phase. This is an interesting initial system with promising demonstrations for contact localization; however to be generally applicable across a wide range of manipulation tasks (esp ones that are contact rich), several issues need further addressal. For these reasons, I will score this a \"weak accept\".", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would like to see the questions I raised in the \"weaknesses\" section of my review discussed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\nThis paper develops an interesting question to the challenging problem of estimating extrinsic contacts solely from images. The paper is very well-written, easy to follow, and describes the method in sufficient detail to allow a skilled practitioner to re-implement it. As such, it is extremely relevant to the robot learning, and particularly, the robot manipulation communities.\n\nA lot of care is taken to ensure the method generalizes to real-world scenarios with non-trivial extrinsic contact. While a simulation environment allows for generating as much synthetic data as possible, the paper demonstrates that training a single model on simulated RGB-D images alone does very poorly when applied in the real world. Much of the design decisions surrounding im2contact are substantiated by the experimental results in Fig. 4 and the tables on the supplemental webpage.\n\nThe qualitative results include a number of interesting scenarios and a set of complex objects. This also includes an out-of-distribution set (although I'd presume this is cherry-picked), of the approach showing some transferability to deformable objects, or rigid objects with highly non-convex shapes.\n\nOverall, the central idea is sound; the experiments are well-executed; and the paper is well-written.\n\n\n## Weaknesses\n\nThere are, however, a few edits I would like to see in the paper, which should further improve its readability; and make it self-contained.\n\n**Discrepancies between Fig. 4 and suppl. tables**: The precision-recall plots in Fig. 4 of the paper seem to have slightly different numbers compared to the tables at the bottom of the supplemental webpage. While the trends are still consistent across them, it will be important to know what causes this discrepancy (perhaps that the experiments each use a different random seed?) Further, moving the real-data table to the main manuscript will make the paper self-contained.\n\n**Impact of optical flow**: The significance of using the optical flow has not been sufficiently established in the main manuscript. The supplemental webpage presents additional tables where the variant w/o flow is worse than the full method (im2contact); but since this is claimed as being essential to improve performance, I would rather have preferred to see this table in the main paper. I would also have liked to see a more nuanced discussion into how optical flow helps (i.e., what kinds of scenarios), and if it hurts in some cases. (currently Fig. 6 is the only anecdotal evidence in support of using flow information)\n\n**Discussion of limitations / assumptions**: im2contact presents an interesting solution to extrinsic contact localization; and the paper thoughtfully discusses some of the practical limitations (contact localization works only in 2D; is noisy; vision-only). A more fundamental assumption underlying im2contact is that extrinsic contacts aren't heavily occluded. However, in practical contact-rich manipulation scenarios (e.g., peg / cable insertion), the grasped object is often self-occluded by the manipulator, which makes 'obj-ref' difficult to specify; limiting the applicability of im2contact.\n\n**Alternative obj-ref schemes**: The design space for specifying the reference object seems a bit under-explored. Currently, 'obj-ref' is an additional reference depth image of the grasped object. Are there any exclusive benefits to using this representation, opposed to alternative choices (e.g., heatmap, region of interest, bounding box, etc.)?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary of Paper
The paper proposes a method for detecting contacts purely based on RGB-D input. The approach is shown to work on real robot data even with occlusions, with various contact types and objects.

## Summary of Reviews
The paper is well written with a compelling idea and method. The experiments are very well executed. Some details needed to be clarified. There were some doubts about the performance. The major concern of several reviewers circled about the question if this method is useful in practice.

## Influence of Rebuttal
The authors nicely argued for why the method is practically relevant. The additional experiments (human video & F/T inclusion) were well received. The quality of the results was well explained.
We had some discussions between the reviewers and AC. In the end also Reviewer rgiT was convinced that the paper is a good fir for CoRL (see updated review). We now see potential application fields for vision-only contact detection in robotics, possibilities to use this method in sensor fusion, we don't think the many false positives are a deal-breaker.

## Suggestions for Improvement
Please include all the promised changes and the new results in the final version. In particular please revise the (by your own admission poorly worded) statements about contact sensing in robotics.
Decision: Accept (Poster)