Active 3D Shape Reconstruction from Vision and Touch

1 Introduction . 3D shape understanding is an active area of research , whose goal is to build 3D models of objects and environments from limited sensory data . It is commonly tackled by leveraging partial observations such as a single view RGB image [ 60 , 67 , 23 , 44 ] , multiple view RGB images [ 16 , 25 , 30 , 31 ] , depth maps [ 61 , 74 ] or tactile readings [ 7 , 49 , 59 , 45 , 38 ] . Most of this research focuses on building shape reconstruction models from a fixed set of partial observations . However , this constraint is relaxed in the active sensing scenario , where additional observation can be acquired to improve the quality of the 3D reconstructions . In active vision [ 4 ] , for instance , the objective can be to iteratively select camera perspectives from an object that result in the highest improvement in quality of the reconstruction [ 76 ] and only very recently the research community has started to leverage large scale datasets to learn exploration strategies that generalize to unseen objects [ 77 , 5 , 41 , 51 , 29 , 78 , 50 , 6 ] . Human haptic exploration of objects both with and without the presence of vision has classically been analysed from a psychological perspective , where it was discovered that the developed tactile exploration strategies for object understanding were demonstrated to not only be ubiquitous , but also highly tailored to specific tasks [ 37 , 33 ] . In spite of this , deep learning-based data-driven approaches to active touch for shape understanding are practically non-existent . Previous haptic exploration works consider objects ∗Correspondence to : ejsmith @ fb.com and edward.smith @ mail.mcgill.ca †Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . independently , and build uncertainty estimates over point clouds , produced by densely touching the objects surface with point-based touch sensors [ 7 , 73 , 28 , 18 , 46 ] . These methods do not make use of learned object priors , and a large number of touches sampled on an object ’ s surface ( over 100 ) is necessary to produce not only a prediction of the surface but also to drive exploration . However , accurate estimates of object shape have also been successfully produced with orders of magnitude fewer touch signals , by making use of high resolution tactile sensors such as [ 75 ] , large datasets of static 3D shape data , and deep learning – see e.g . [ 56 , 68 ] . Note that no prior work exists to learn touch exploration by leveraging large scale datasets . Moreover , no prior work explores active touch in the presence of visual inputs either ( e.g . an RGB camera ) . Combining the recent emergence of both data-driven reconstruction models from vision and touch systems [ 56 , 68 ] , and data-driven active vision approaches , we present a novel formulation for active touch exploration . Our formulation is designed to easily enable the use of vision signals to guide the touch exploration . First , we define a new problem setting over active touch for 3D shape reconstruction where touch exploration strategies can be learned over shape predictions from a learned reconstruction model with strong object priors . Second , we develop a simulator which allows for fast and realistic grasping of objects , and for extracting both vision and touch signals using a robotic hand augmented with high-resolution tactile sensors [ 75 ] . Third , we present a 3D reconstruction model from vision and touch which produces mesh-based predictions and achieves impressive performance in the single view image setting , both with and without the presence of touch signals . Fourth , we combine the simulator and reconstruction model to produce a tactile active sensing environment for training and evaluating touch exploration policies . The outline for this environment can be viewed in Figure 1 . Over the provided environment , we present a series of datadriven touch exploration models that take as an input a mesh- based shape reconstruction and decide the position of the next touch . By leveraging a large scale dataset of over 25k CAD models from the ABC dataset [ 34 ] together with our environment , the data-driven touch exploration models outperform baseline policies which fail to leverage learned patterns between object shape or the distribution of object shapes and optimal actions . We demonstrate our proposed data-driven solutions perform up to 18 % better than random baselines and lead to impressive object reconstructions relative to their input modality , such as those demonstrated in Figure 2 . Our framework , training and evaluation setup , and trained models are publicly available on a GitHub repository to ensure and encourage reproducible experimental comparison 3 . 3https : //github.com/facebookresearch/Active-3D-Vision-and-Touch 2 Related Work . 3D reconstruction . 3D shape reconstruction is a popular and well studied area of research with a multitude of approaches proposed across varying 3D representations [ 13 , 63 , 19 , 52 , 67 , 23 , 12 , 71 ] and input modalities [ 27 , 53 , 69 , 57 ] . While still a niche topic , a number of works have attempted to reconstruct 3D shape from only touch signals [ 7 , 49 , 59 , 45 , 38 ] . These works all assume point based touch signals which supply at most one point of contact of normal information . A limited number of methods have also been proposed for 3D reconstruction from both vision and touch signals [ 9 , 26 , 22 , 68 , 69 , 56 ] . In contrast to these works which use static datasets , we focus on improving 3D reconstruction through tactile exploration . Shape from interaction methods have also been proposed for object reconstruction through hand interactions [ 42 , 64 , 47 ] , though in contrast to our work an image of the interaction provides the additional information rather than touch readings . Active sensing for reconstruction . Active vision aims to manipulate the viewpoint of a camera in order to maximize the information for a particular task [ 11 , 76 ] . Similar to our setting , active vision can be useful in 3D shape reconstruction [ 62 , 70 , 17 , 41 , 14 , 35 ] , but where camera perspectives are planned instead of grasp locations . Only very recently have deep learning active vision approaches been proposed for 3D object reconstruction [ 77 , 5 , 41 , 51 ] . Deep learning based active sensing for reconstruction has also recently emerged in the medical imaging domain , where the time spent performing MRI scans has been reduced by learning to select a small number of more informative frequencies over a pre-trained reconstruction model [ 29 , 78 , 50 , 6 ] . Touch-based active exploration for shape understanding . Initial works tackling active tactile exploration focused on object recognition . [ 1–3 ] . While some of these methods do integrate both vision and touch for active shape understanding , they only focus on object recognition , operate over small datasets of already known and observed objects and do not leverage deep learning tools for learning patterns over large datasets of object shapes . Fleer and Moringen et . al . proposed to learn haptic exploration strategies using a reinforcement learning framework over a recurrent attention model , however these strategies were optimized for object classification and only trained over a dataset of 4 objects with a single , floating , depth-based tactile sensor array [ 20 ] . Several priors works have explored the problem of active acquisition of touch signals specifically for for surface reconstruction . A common theme among most approaches is to predict object shape using Gaussian processes so that uncertainty estimates are naturally available to drive the selection of the next point to touch , which is performed using a single finger [ 28 , 73 , 18 ] . To improve the speed of shape estimation during tactile exploration Matsubara et . al . [ 40 ] consider both uncertainty and travel cost when selecting touches using graph-based path planning . Bierbaum et al . [ 8 ] instead drove exploration through a dynamic potential field approach made popular in robot navigation to produce point cloud predictions , and extracted touch signals using a comparatively unrealistic 5 finger robotic hand model in simulation . These methods all make use of point based tactile sensors and use deterministic strategies , which are tuned and evaluated over a very small number of objects. . In contrast to these works , our approach is fully data driven , and as such , actions are selected using policies trained and evaluated over tens of thousands of objects . Moreover , predictions are made in mesh space fusing visual information with high resolution tactile signals extracted from objects in simulation using a realistic robot hand . 3 Active touch exploration . In our proposed active touch exploration problem , given a pre-trained shape reconstruction model over touch and optionally vision signals , the objective is to select the sequence of touch inputs which lead to the highest improvement in reconstruction accuracy . To tackle this problem , we define an active touch environment that contains a simulator , a reconstruction model ( a pre-trained neural network ) , and a loss function . The simulator takes as input a 3D object shape O together with parameters describing a grasp , g , and outputs touch readings , t , of the 3D shape at the grasp locations along with an RGB image of the object , I . The reconstruction model is a neural network parametrized by ϕ , which takes an input X and produces the current 3D shape estimate Ô as follows : Ô = f ( X ; ϕ ) . In our setup , we investigate two reconstruction model variants which differ in their inputs : 1 ) the model only receives a set of touch readings , t , such that X = t , and 2 ) the model receives both a set of touch readings , t , and an RGB image rendering of the shape , I , such that X = { t , I } . The loss function takes as input the current belief of the object ’ s shape , Ô , and the ground truth shape , O , and computes the distance between them : d ( O , Ô ) . Thus , active touch exploration can be formulated as sequentially selecting the optimal set of K grasp parameters { g1 , g2 ... , gK } that maximize the similarity between the ground truth shape O and the reconstruction output after K grasps ÔK 4 : argmin g1 , g2 , .... gK d ( O , ÔK ) , ( 1 ) where gk determine the touch readings fed to the reconstruction network producing Ôk . We use the Chamfer distance ( CD ) [ 60 ] between the predicted and target surface as the distance metric in our active touch formulation . Further details with respect to the CD are provided in the supplemental materials . In the reminder of this section , we provide details about the touch simulator and the reconstruction model . 3.1 Active touch simulator . Conceptually , our simulator can be described in the five steps depicted in Figure 3 . First , an object is loaded onto the environment . Second , an action space is defined around the 3D object by uniformly placing 50 points on a sphere centered at the center of the loaded object . Third , to choose a grasp , one of the points is selected and a 4-digit robot hand is placed such that its 3rd digit lies on the point and the hand ’ s palm lies tangent to the sphere . Fourth , the hand is moved towards the center of the object until it comes in contact with it . Last , the fingers of the hand are closed until they reach the maximum joint angle , or are halted by the contact with the object . As a result , the simulator produces 4 touch readings ( one from each finger of the hand ) and one RGB image of the object . Note that each action is defined by its position index on the sphere of 50 actions . This parameterization is selected specifically as it does not require any prior knowledge of the object , other than its center , and in simulation it consistently leads to successful interactions between the hand ’ s touch sensors and the object ’ s surface . In our simulator , all steps are performed in python across the robotics simulator PyBullet [ 15 ] , the rendering tool Pyrender [ 39 ] , and PyTorch [ 48 ] . For a given grasp and object , the object is loaded into PyBullet [ 15 ] , along with a Wonik ’ s Allegro Hand [ 54 ] equipped with vision-based touch sensors [ 36 ] on each of its fingers , and then the point in space corresponding to the action to be performed is selected and the grasping procedure is performed using PyBullet ’ s physics simulator . Pose information from the produced grasps is then extracted and used by Pyrender to render both a depth map of the object from the perspective of each sensor and an RGB image of the object from a fixed perspective . The depth maps are then converted into simulated touch signals using the method described in [ 56 ] . All steps in this procedure are performed in parallel or using GPU accelerated computing , and as a result across the 50 grasping options of 100 randomly chosen objects , simulated grasps and touch signals are produced in ∼ 0.0317 seconds each on a Tesla V100 GPU with 16 CPU cores . Our simulator supports two modes of tactile exploration grasping and poking . In the grasping scenario , the hand is performing a full grasp of an object using all four fingers . While , in the poking scenario , only the index finger of the hand is used for touch sensing . Further details on this simulated environment are provided the supplemental materials .

Summary: This paper introduces and explores the task of active 3D reconstruction using visual and haptic data. Instead of relying on static data to perform reconstruction, the learning process is guided by exploration, from which new data can be accounted for in the reconstruction process by the proposed model. In order to enable this setting/task, the authors develop a simulator that provides both visual and touch signals that are fed to the model, and an environment for training and evaluating policies.