ReSkin: versatile, replaceable, lasting tactile skins
Authors: Raunaq Bhirangi
Tess Hellebrekers
Carmel Majidi
Abhinav Gupta
Venue: CoRL2021 Oral (2021)
Keywords: Tactile Skin
Self-supervised Learning
Magnetic Sensing
Soft Sensors

Abstract: Soft sensors have continued growing interest in robotics, due to their ability to enable both passive conformal contact from the material properties and active contact data from the sensor properties. However, the same properties of conformal contact result in faster deterioration of soft sensors and larger variations in their response characteristics over time and across samples, inhibiting their ability to be long-lasting and replaceable. ReSkin is a tactile soft sensor that leverages machine learning and magnetic sensing to offer a low-cost, diverse and compact solution for long-term use. Magnetic sensing separates the electronic circuitry from the passive interface, making it easier to replace interfaces as they wear out while allowing for a wide variety of form factors. Machine learning allows us to learn sensor response models that are robust to variations across fabrication and time, and our self-supervised learning algorithm enables finer performance enhancement with small, inexpensive data collection procedures. We believe that ReSkin opens the door to more versatile, scalable and inexpensive tactile sensation modules than existing alternatives. https://reskin.dev


Reviews:
[{"review_id": "pI1QHiHWo5", "reviewer": "Reviewer_2SmQ", "summary": "In this paper, the authors present a new type of low cost contact sensor based on magnetic field distortions. The core idea is to use fabricated elastomer materials with embedded magnetic particles. When the material is distorted under pressure, the relative distances between the magnetic particles will also shift. The change in the magnetic flux can be detected by an array of magnetometers. The authors used supervised training to calibrate the contact sensor, i.e. map magnetic field readings to contact location and force magnitude using a neural network. They also proposed a triplet loss to adapt their learned sensor response model to changes in the elastomer material or sensor boards without the need to use labelled new data.  The authors demonstrated that their contact sensor can be used for various applications such as manipulation and locomotion. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper includes:\n\n(1) A fabrication process to produce cheap, replaceable \"skins\" that can be used to measure contacts.\n\n(2) A learned sensor response model that maps the magnetic flux to contact forces and locations. By combining readings from multiple sensors and regularizing the training using a self-supervision loss, the model can generalize to new materials and sensor boards without complicated calibration process. \n\n(3) Can be applied to many different types of tasks. \n\nThe main limitations of the paper are:\n\n(1) Generalization to simultaneously multiple contact points. In the proposed pipeline, the authors used single point contact data (i.e.  \"indentation\") to train their model. And the validation and adaptation also depends on point contact responses. This raised a question on whether/why their learned model can generalize to a multi-point contact, or surface contact situations. The deformation of the material is a nonlinear response, so one cannot just assume a simple addition of the magnetic flux happens under a continuous surface load. Needs more discussion and experiments to explain and validate the more general situation. For example, Is a ReSkin unit really measuring the center of pressure or something else?\n\n(2) Continuous adaptation. The authors claimed that when a new \"skin\" is replaced, the model can still generalize by self-supervised training using the triplet loss + old labelled data. The question is if this process will eventually produce bias overtime? For example, what happens if one swaps the elastomer 10 times? After 10 adaptations is there any \"drift\" in the model so the performance will degrade? Needs more discussion and experiments on this.\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given the broad potential application and simplicity of recalibration using self-supervision, I think the ReSkin technique and their calibration model can be potentially very useful for the robotics community in general. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See my comments in the main limitations. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "fmcmMvTSrqn", "reviewer": "Reviewer_VwcU", "summary": "The paper proposes a new soft tactile sensor that relies on the change in the magnetic properties of the soft magnetized skin to estimate and localize the applied force. Experiments are carried out to learn the models that map the magnetic field measurements to applied forces, and techniques for generalization when the skin or PCB is replaced are proposed and evaluated. Example applications are described in the paper and demonstrated in the video.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed sensor has a number of desirable properties: it is small, durable, allows for easy replacement of skin, etc.\n\nWeaknesses: Experiments only seem to evaluate the normal force. Is the estimation of the shear force also reliable? Can one estimate force when the contact surface is large? Can one estimate the skin deformation from the magnetic flux readings?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed sensor design may be potentially useful in many applications. The paper showed in detail that the sensor can be utilized at high rate (roughly 400Hz) and with good precision (roughly 1mm). The fact that the sensor design and code are publicly released is a big bonus.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The video can be shortened. Picking grapes and blueberries is roughly the same, perhaps one of the experiments can be dropped from the video. It would be good to see the sensor readings when the berries are picked up. How noisy are those measurements?\n\nWhat happens if there is external magnetic field? Is the sensor not useful then anymore? It may be worth adding a paragraph or subsection discussing the limitations of the current design. For example, if one manipulates metal objects that can be magnetized, will this affect the performance? Can you suggest any solutions for this problem?"}}, {"review_id": "VfsBoPfhCKA", "reviewer": "Reviewer_fsSv", "summary": "The work presents a tactile sensor named as ReSkin that can replaces its skin. It is based on magnetic sensing that separates the electronic circuitry from the passive interface, making it easier to replace interfaces as they wear out. A neural network method is also proposed to to learn sensor response models that are variant to across fabrication and time.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The presented replaceable skin and the model to learn sensor responses across fabrication and time are interesting.\n2. Some interesting results are presented in the experiments.\n\nWeaknesses:\nThe contributions of the paper are not clear as some of the available tactile sensors come with replaceable skins naturally, for example GelSight and GelTip sensors. It is not very convincing to claim that a tactile sensor with replaceable skin is proposed. Also, it is clear how the proposed sensor is different from the magnetic sensors in the literature. It is interesting to learn sensor responses across fabrication and time are interesting but it can be taken separately from the design and fabrication of the ReSkin.\nGomes, D.F., Lin, Z. and Luo, S., 2020, June. GelTip: A finger-shaped optical tactile sensor for robotic manipulation. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9903-9909). IEEE.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Magnetic sensors are one major type of the tactile sensors for robotic applications, it is not clear how the proposed sensor differs from the ones in the literature. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The presentation of the paper can be improved, for example,\n1. Experiment Setup is ahead of the sensor models, which makes it confusing to follow.\n2. In Section 5, the network architecture can be better illustrated.\n3. The contact localization is a easy task for the tactile sensor as it is to detect the contact on the surface, it would be better to include analysis on adaptation for force prediction."}}, {"review_id": "IZa2oJFyz1M", "reviewer": "Reviewer_51C5", "summary": "The paper proposes ReSkin, an inexpensive, replaceable, and robust tactile skin based on magnetometer-based sensing mechanism and discusses learning methods for calibrating the sensor raw signals to contact localization and force prediction output across different skins and sensing boards.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The skin is inexpensive, replaceable, and robust to wear and tear. I really like how easy it is to replace the skin layer\n- The skin can sense normal forces and potentially shear forces as well (not shown in this paper)\n- The evaluation was done with a large variety of skins\n- The demonstrations were shown on a variety of applications\n\nSuggestions for improvement:\n- Clarifying contributions with respect to the state-of-the-art: There is an extensive body of work on tactile skins (both large-area and small with resistive, capacitive, magnetic, optical, and multimodal sensing etc.) for robots which have not been cited in this paper. Clarifying the contributions of this paper with respect to this extensive body of work (especially large-area skins which was one of the primary motivations of this paper ( https://spj.sciencemag.org/journals/research/2019/3018568/, https://www.sciencedirect.com/science/article/pii/S0921889014001821,  https://ieeexplore.ieee.org/document/6548392, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8812712, https://www.sciencedirect.com/science/article/abs/pii/S0921889016307837, https://arxiv.org/abs/1803.00628, https://link.springer.com/chapter/10.1007/978-3-319-04123-0_6, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9247533 , and many more ...) and other magnetic sensing based skins that the authors have already cited such as Hellebrekers et al.,Yan et al.) would make the paper stronger.\n\n- The paper can also become stronger by discussing the limitations of this sensing skin. Is the sensing technology affected by the presence of other magnetic objects in the environment? Are multiple-touch scenarios possible with this skin (all the demonstrations and data collection were single touch)? Is there any cross-talk when multiple touch points are proximally located? While the data acquisition rate is impressive, the latency is not clear. Analyzing these effects or discussing limitations and potential solutions can make the paper more complete. \n\n- While the evaluation was done with a variety of skins and a lot of data, it is not clear what the insights from the generalization results are. The paper can benefit from details such as how do the 18 different sensing skins differ so that it’s clear what is the generalization across? Do the results generalize across different indenter sizes? In touch modality, sensing is inherently coupled to action and analyzing these effects while considering variations across parameters that affect the actions are crucial. While the results with self-supervised adaptation across multiple sensors are encouraging, in a new scenario, the fine-tuning demonstrated in the paper is done using data collected in a very controlled scenario. Is this feasible in unstructured real-world scenarios?\n\n- Has the paper considered using model-based approaches for calibration and using data-driven methods to model the residuals or unmodeled effects? In the real-world, collecting data is hard especially with contact and conditions vary widely. Having a physics-based model as a prior or to provide some structure can potentially benefit a lot during calibration in previously unseen scenarios.\n\n- Can the paper discuss how (or if) the sensing method can be extended to normal 6D force/torque sensing and shear sensing, which could be very useful in real physical interactions? This could be a strong differentiating factor with many existing large-area tactile sensing skins.\n\n- The paper should be carefully reviewed throughout for typos and word omissions. For example: Line 254: … data, and ?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper discusses an encouraging tactile skin technology and learning-based calibration method, it is not clear what the contributions are with respect to other magnetic skins or large-area skins in general. Also, the applicability of the skin and scalability of the calibration procedure needs to be analyzed especially in the light of unstructured scenarios. This is on the way to becoming a strong paper but it needs more work at its current state.\n\n------\n\nThe rebuttal has addressed most of my concerns. I am changing my score to Weak Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See “Suggestions for Improvement” section above.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a magnetic, replaceable soft skin sensor with novel form factor. The paper explains the design for improving and producing more skin sensors for use in robotic manipulation with a specific focus towards enabling the use of learning-based methods. The paper uses learning to create sensor processing that is robust across the variations of different manufactured skin elements.

Furthermore the authors argue for the need of tactile sensing as a common modality in robot manipulation to improve performance when working outside of structured environments. Furthermore, they explain why learning is well suited for use with tactile sensors.

I agree with this importance and the recent growth in tactile sensing for learning-based manipulation provides further evidence for its importance. As such, I think this paper would be an interesting inclusion at CoRL and spark helpful discussions.

Finally, I agree with reviewer 51C5 and caution the authors against overly broad claims such as "But above all, we believe our sensor is the only sensor which satisfies all the requirements to emerge as the only viable sensor for learning applications." Indeed the citations being given show other sensors are viable for learning. Perhaps the authors mean that the durability and replacement of the sensors make them more suitable to long term use and abuse that is common in reinforcement learning techniques. However, between safe RL methods and other more careful data-collection used in previously published learning-based work for tactile sensing, other sensors are clearly appropriate for robot learning.

I believe the paper would be better received and appreciated if the authors offer this as an additional sensor that can often be useful on its own, but can also complement other existing sensors (e.g. I could imagine using BioTacs or GelSight sensors on the fingertips of a robot with the ReSkin covering the rest of the robot hand and arm.


Decision: Accept (Oral)