Active 3D Shape Reconstruction from Vision and Touch

1 Introduction . 3D shape understanding is an active area of research , whose goal is to build 3D models of objects and environments from limited sensory data . It is commonly tackled by leveraging partial observations such as a single view RGB image [ 60 , 67 , 23 , 44 ] , multiple view RGB images [ 16 , 25 , 30 , 31 ] , depth maps [ 61 , 74 ] or tactile readings [ 7 , 49 , 59 , 45 , 38 ] . Most of this research focuses on building shape reconstruction models from a fixed set of partial observations . However , this constraint is relaxed in the active sensing scenario , where additional observation can be acquired to improve the quality of the 3D reconstructions . In active vision [ 4 ] , for instance , the objective can be to iteratively select camera perspectives from an object that result in the highest improvement in quality of the reconstruction [ 76 ] and only very recently the research community has started to leverage large scale datasets to learn exploration strategies that generalize to unseen objects [ 77 , 5 , 41 , 51 , 29 , 78 , 50 , 6 ] . Human haptic exploration of objects both with and without the presence of vision has classically been analysed from a psychological perspective , where it was discovered that the developed tactile exploration strategies for object understanding were demonstrated to not only be ubiquitous , but also highly tailored to specific tasks [ 37 , 33 ] . In spite of this , deep learning-based data-driven approaches to active touch for shape understanding are practically non-existent . Previous haptic exploration works consider objects ∗Correspondence to : ejsmith @ fb.com and edward.smith @ mail.mcgill.ca †Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . independently , and build uncertainty estimates over point clouds , produced by densely touching the objects surface with point-based touch sensors [ 7 , 73 , 28 , 18 , 46 ] . These methods do not make use of learned object priors , and a large number of touches sampled on an object ’ s surface ( over 100 ) is necessary to produce not only a prediction of the surface but also to drive exploration . However , accurate estimates of object shape have also been successfully produced with orders of magnitude fewer touch signals , by making use of high resolution tactile sensors such as [ 75 ] , large datasets of static 3D shape data , and deep learning – see e.g . [ 56 , 68 ] . Note that no prior work exists to learn touch exploration by leveraging large scale datasets . Moreover , no prior work explores active touch in the presence of visual inputs either ( e.g . an RGB camera ) . Combining the recent emergence of both data-driven reconstruction models from vision and touch systems [ 56 , 68 ] , and data-driven active vision approaches , we present a novel formulation for active touch exploration . Our formulation is designed to easily enable the use of vision signals to guide the touch exploration . First , we define a new problem setting over active touch for 3D shape reconstruction where touch exploration strategies can be learned over shape predictions from a learned reconstruction model with strong object priors . Second , we develop a simulator which allows for fast and realistic grasping of objects , and for extracting both vision and touch signals using a robotic hand augmented with high-resolution tactile sensors [ 75 ] . Third , we present a 3D reconstruction model from vision and touch which produces mesh-based predictions and achieves impressive performance in the single view image setting , both with and without the presence of touch signals . Fourth , we combine the simulator and reconstruction model to produce a tactile active sensing environment for training and evaluating touch exploration policies . The outline for this environment can be viewed in Figure 1 . Over the provided environment , we present a series of datadriven touch exploration models that take as an input a mesh- based shape reconstruction and decide the position of the next touch . By leveraging a large scale dataset of over 25k CAD models from the ABC dataset [ 34 ] together with our environment , the data-driven touch exploration models outperform baseline policies which fail to leverage learned patterns between object shape or the distribution of object shapes and optimal actions . We demonstrate our proposed data-driven solutions perform up to 18 % better than random baselines and lead to impressive object reconstructions relative to their input modality , such as those demonstrated in Figure 2 . Our framework , training and evaluation setup , and trained models are publicly available on a GitHub repository to ensure and encourage reproducible experimental comparison 3 . 3https : //github.com/facebookresearch/Active-3D-Vision-and-Touch 2 Related Work . 3D reconstruction . 3D shape reconstruction is a popular and well studied area of research with a multitude of approaches proposed across varying 3D representations [ 13 , 63 , 19 , 52 , 67 , 23 , 12 , 71 ] and input modalities [ 27 , 53 , 69 , 57 ] . While still a niche topic , a number of works have attempted to reconstruct 3D shape from only touch signals [ 7 , 49 , 59 , 45 , 38 ] . These works all assume point based touch signals which supply at most one point of contact of normal information . A limited number of methods have also been proposed for 3D reconstruction from both vision and touch signals [ 9 , 26 , 22 , 68 , 69 , 56 ] . In contrast to these works which use static datasets , we focus on improving 3D reconstruction through tactile exploration . Shape from interaction methods have also been proposed for object reconstruction through hand interactions [ 42 , 64 , 47 ] , though in contrast to our work an image of the interaction provides the additional information rather than touch readings . Active sensing for reconstruction . Active vision aims to manipulate the viewpoint of a camera in order to maximize the information for a particular task [ 11 , 76 ] . Similar to our setting , active vision can be useful in 3D shape reconstruction [ 62 , 70 , 17 , 41 , 14 , 35 ] , but where camera perspectives are planned instead of grasp locations . Only very recently have deep learning active vision approaches been proposed for 3D object reconstruction [ 77 , 5 , 41 , 51 ] . Deep learning based active sensing for reconstruction has also recently emerged in the medical imaging domain , where the time spent performing MRI scans has been reduced by learning to select a small number of more informative frequencies over a pre-trained reconstruction model [ 29 , 78 , 50 , 6 ] . Touch-based active exploration for shape understanding . Initial works tackling active tactile exploration focused on object recognition . [ 1–3 ] . While some of these methods do integrate both vision and touch for active shape understanding , they only focus on object recognition , operate over small datasets of already known and observed objects and do not leverage deep learning tools for learning patterns over large datasets of object shapes . Fleer and Moringen et . al . proposed to learn haptic exploration strategies using a reinforcement learning framework over a recurrent attention model , however these strategies were optimized for object classification and only trained over a dataset of 4 objects with a single , floating , depth-based tactile sensor array [ 20 ] . Several priors works have explored the problem of active acquisition of touch signals specifically for for surface reconstruction . A common theme among most approaches is to predict object shape using Gaussian processes so that uncertainty estimates are naturally available to drive the selection of the next point to touch , which is performed using a single finger [ 28 , 73 , 18 ] . To improve the speed of shape estimation during tactile exploration Matsubara et . al . [ 40 ] consider both uncertainty and travel cost when selecting touches using graph-based path planning . Bierbaum et al . [ 8 ] instead drove exploration through a dynamic potential field approach made popular in robot navigation to produce point cloud predictions , and extracted touch signals using a comparatively unrealistic 5 finger robotic hand model in simulation . These methods all make use of point based tactile sensors and use deterministic strategies , which are tuned and evaluated over a very small number of objects. . In contrast to these works , our approach is fully data driven , and as such , actions are selected using policies trained and evaluated over tens of thousands of objects . Moreover , predictions are made in mesh space fusing visual information with high resolution tactile signals extracted from objects in simulation using a realistic robot hand . 3 Active touch exploration . In our proposed active touch exploration problem , given a pre-trained shape reconstruction model over touch and optionally vision signals , the objective is to select the sequence of touch inputs which lead to the highest improvement in reconstruction accuracy . To tackle this problem , we define an active touch environment that contains a simulator , a reconstruction model ( a pre-trained neural network ) , and a loss function . The simulator takes as input a 3D object shape O together with parameters describing a grasp , g , and outputs touch readings , t , of the 3D shape at the grasp locations along with an RGB image of the object , I . The reconstruction model is a neural network parametrized by ϕ , which takes an input X and produces the current 3D shape estimate Ô as follows : Ô = f ( X ; ϕ ) . In our setup , we investigate two reconstruction model variants which differ in their inputs : 1 ) the model only receives a set of touch readings , t , such that X = t , and 2 ) the model receives both a set of touch readings , t , and an RGB image rendering of the shape , I , such that X = { t , I } . The loss function takes as input the current belief of the object ’ s shape , Ô , and the ground truth shape , O , and computes the distance between them : d ( O , Ô ) . Thus , active touch exploration can be formulated as sequentially selecting the optimal set of K grasp parameters { g1 , g2 ... , gK } that maximize the similarity between the ground truth shape O and the reconstruction output after K grasps ÔK 4 : argmin g1 , g2 , .... gK d ( O , ÔK ) , ( 1 ) where gk determine the touch readings fed to the reconstruction network producing Ôk . We use the Chamfer distance ( CD ) [ 60 ] between the predicted and target surface as the distance metric in our active touch formulation . Further details with respect to the CD are provided in the supplemental materials . In the reminder of this section , we provide details about the touch simulator and the reconstruction model . 3.1 Active touch simulator . Conceptually , our simulator can be described in the five steps depicted in Figure 3 . First , an object is loaded onto the environment . Second , an action space is defined around the 3D object by uniformly placing 50 points on a sphere centered at the center of the loaded object . Third , to choose a grasp , one of the points is selected and a 4-digit robot hand is placed such that its 3rd digit lies on the point and the hand ’ s palm lies tangent to the sphere . Fourth , the hand is moved towards the center of the object until it comes in contact with it . Last , the fingers of the hand are closed until they reach the maximum joint angle , or are halted by the contact with the object . As a result , the simulator produces 4 touch readings ( one from each finger of the hand ) and one RGB image of the object . Note that each action is defined by its position index on the sphere of 50 actions . This parameterization is selected specifically as it does not require any prior knowledge of the object , other than its center , and in simulation it consistently leads to successful interactions between the hand ’ s touch sensors and the object ’ s surface . In our simulator , all steps are performed in python across the robotics simulator PyBullet [ 15 ] , the rendering tool Pyrender [ 39 ] , and PyTorch [ 48 ] . For a given grasp and object , the object is loaded into PyBullet [ 15 ] , along with a Wonik ’ s Allegro Hand [ 54 ] equipped with vision-based touch sensors [ 36 ] on each of its fingers , and then the point in space corresponding to the action to be performed is selected and the grasping procedure is performed using PyBullet ’ s physics simulator . Pose information from the produced grasps is then extracted and used by Pyrender to render both a depth map of the object from the perspective of each sensor and an RGB image of the object from a fixed perspective . The depth maps are then converted into simulated touch signals using the method described in [ 56 ] . All steps in this procedure are performed in parallel or using GPU accelerated computing , and as a result across the 50 grasping options of 100 randomly chosen objects , simulated grasps and touch signals are produced in ∼ 0.0317 seconds each on a Tesla V100 GPU with 16 CPU cores . Our simulator supports two modes of tactile exploration grasping and poking . In the grasping scenario , the hand is performing a full grasp of an object using all four fingers . While , in the poking scenario , only the index finger of the hand is used for touch sensing . Further details on this simulated environment are provided the supplemental materials .

Summary: This paper presents an approach for active 3D reconstruction of objects based solely on touch, and optionally together with vision. The main idea is active perception -- estimate where to touch/see the object in order to better reconstruct it. The proposed method to solve this involves a neural network that fuses touch and vision signals to estimate 3D shape, and a policy that predicts where to touch next for best reconstruction performance. Data comes from a simulator that produces both touch and image signals (from the ABC dataset).