Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control
Authors: Miquel Oller
Mireia Planas i Lisbona
Dmitry Berenson
Nima Fazeli
Venue: CoRL 2022 Poster (2022)
Keywords: manipulation
tactile control
deformable tactile sensors

Abstract: Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasped object poses are recovered from membrane geometry and reaction wrenches, decoupling interaction dynamics from the tactile observation model. We benchmark our approach on two real-world contact-rich tasks: drawing with a grasped marker and in-hand pivoting. Our results suggest that explicitly modeling membrane dynamics achieves better task performance and generalization to unseen objects than baselines.


Reviews:
[{"review_id": "n7NDMHnEMpu", "reviewer": "Reviewer_xvAZ", "summary": "The main contribution of the paper is a framework to learn a membrane dynamics model for their custom bubble membrane fingertips for use in tasks such as continuously pressing down on a whiteboard with a marker to draw a line and reorienting an object between the fingers by exploiting the environment (a table). There is a time of flight depth sensor inside of the bubble gripper that can generate a pointcloud of the surface of the gripper. With the depth information, they can use PointNet and other encoders and decoders to generate embeddings of objects that the bubble gripper is grasping. They can then learn the membrane dynamics model that can then be used to complete the 2 above tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I really liked how the authors compared the sensor deformation between 2 sensors they had available. It helps to provide some context between the size and compliance of multiple sensors instead of only reporting the results of one sensors. I do know that this Bubble gripper has already been introduced in RoboSoft so this is not a hardware paper.\nThe limitations of the method and bubble gripper were also adequately raised in the paper and video so I don't have any concerns. \nI also did like the overlay of the desired orientation of the object in the video as I was at first confused as to what the green line was.\nIf I were to nitpick something, I would have wanted to see if by using the gelsight slim and a 2D image embedding network, how would the performance compare with the PointNet based network. In addition, I would be curious to know how much data would be needed for each training to have adequate performance or equivalent results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think this paper is useful for understanding how to best use the depth information that the time of flight sensor inside the bubble gripper can be used to help the robot reorient an object for better tool use. I think it is valuable, although not many sensors are similar to the bubble gripper that provide a point cloud and would use PointNet. I feel like for other sensors like GelSight and Fingervision, having the images would be easier to handle than the point clouds, but that's just a difference in sensor types (2D vs 3D vision) so I can't fault it. I thought that the experiments were thorough with a variety of different objects of the similar cylindrical handle used. However, I would have liked if there had been an experiment of using the bubble gripper to use a screwdriver to tighten a screw like was teased in the video, but maybe its just due to the compliance of the bubble gripper that it would be unable to grip it strongly enough in order to produce the torque necessary to tighten. Anyways, I decided on the weak accept because I think that although there is a contribution in the paper, it seems like the method may only be applicable to this sensor unless there comes along a different sensor that also relies on point clouds.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I don't really have any issues with the paper. There were plenty of experiments and the results were presented in a satisfactory method. The paper was also well written and fairly straightforward. Thus, I don't really have any concerns with the paper."}}, {"review_id": "_BduMlboI0_", "reviewer": "Reviewer_rK46", "summary": "The paper proposes a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment.  Authors explore the differences between soft(large deformation) and hard sensors, and demostrate two manipulation tasks with a Soft Bubbles.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.  explore the differences between soft and hard sensors.\n2. propose a method to learn the large deformation sensor dynamics.\n3. benchmark the method on two real-world contact-rich tasks.\n\nWeaknesses:\nI do not see explicit weakness in the paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In manipulation tasks, tactile sensation plays an important role. Comparing to hard sensor, this paper uses a large deformation sensor to collecting rich contact information, and complete contact-rich tasks. \nThe membranes seem like a physical filter, may smooth the object geometry, and filter the details of the objects. How it performs in a fine manipulation tasks, such as recognize two sides of usb flash drive or manipulating a small object?", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": " no explicit issues.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Pq4yxkjINCz", "reviewer": "Reviewer_ke9K", "summary": "I found the paper somewhat opaque, but I think the aim is to model the state of the tactile sensor as a collection of poses on the contact surface, then use model-predictive control (i.e. rollout action trajectories to a goal state) to achieve desired states of the contact surface, and in so doing manipulate held tools (constrained to long thin tools that can be held and pivoted by the gripper). This capability was demonstrated by drawing a line with various shaped markers and pivoting objects to a desired angle.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Interesting combination of tasks. I think the focus towards tool use (which this paper is basically about) is a good direction to take the field.\n- The use of model-predictive control on tactile data is a step forward in the field. \n\nWeaknesses\n- I did find the paper difficult to decode. The paper was generally well written, but perhaps it is coming from a different viewpoint from how I see things. I was even mystified on the first sentence, when I was wondering what 'collocated' meant (and whether it has an extra 'l'...).  New terminology seems to be dropped in willy-nilly, e.g. saying there is localized and distributed tactile sensing (isn't what they call localized tactile sensing just force-torque sensing, which is why 'force and tactile' are usually grouped together?) This was all dropped in without any clear problem statement of what the paper is contributing.\n- I welcome the attempt to compare two distinct tactile sensors (Soft Bubble and GelSlim), but then this just stopped after the first results section 4.1. What is the point of having this comparison if you're not going to use it in the rest of the paper? If throughout the paper you'd compared both sensors (even if one of them does not work well on the tasks) I would have seen the study as a lot stronger. As it was, I was first looking for results with the GelSlim, then wondering if there are none what are the concrete criticisms of the sensor that makes it not worth pursuing (and if so why not just delete the GelSlim from the paper entirely?)\n- I could not understand how your controller works. How are you setting the goal? How does that relate to the deformation of the membrane? How is that set? How does this relate to the tactile data have from the sensor? This was all unclear to me. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this is fundamentally interesting and novel work, but I find the presentation so opaque that I cannot judge for sure.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see comments above.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "4egGIyBREXm", "reviewer": "Reviewer_gpA9", "summary": "The paper proposed a membrane dynamic model of the soft Bubble sensor. The model takes the perceived 3D geometry of the membrane, wrench of manipulator’s end-effector, robot actions, and object geometry to predict future membrane deformations and wrench. A key feature of the model is that the decoupling of the sensor dynamics from the task dynamics so that the membrane dynamic model may generalize to different objects and tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are:\n1.\tThe problem formulation and methods are well presented in words and in mathematical notations that can be easily understood. The supplementary materials are also well organized and include enough implementation details of the learning process.\n2.\tThe main feature of the proposed model is stated clearly and is emphasized and explained in a couple of places, which is helpful for the readers. \n\nThe main weaknesses of the paper are:\n1.\tThe membrane dynamic model is relatively complex combining a variety of sensory data and known information on the object’s geometry, which limits the usage of the proposed model.\n2.\tThe setup of the two real-world tasks are relatively easy compared to what the soft membrane sensor is capable of as described in the paper. The authors might increase the difficulty of the tasks to some degree.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written, and the proposed methods are clearly presented and supported by the hardware experimental results. The paper could be improved by addressing the weaknesses and issues. Hence, I recommend weak accept for the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1.\tConsidering the complexity of the controller, it is of interest to know what is the computational time cost of updating the optimal action, which is an important characteristics for controller?\n2.\tThe authors are suggested to also add experiment of drawing standard circles, which should be more convincing than the simple straight line.\n3.\tIn section 5.2, the experimental results show that the test objects perform even slightly better than the training objects, which is possibly because the test objects are easier to control according to the authors. However, this makes the conclusion on the generalization of the proposed model somehow questionable?\n"}}]

Author rebuttal:

Decision: Accept (poster)