Active 3D Shape Reconstruction from Vision and Touch
Authors: Edward J. Smith
David Meger
Luis Pineda
Roberto Calandra
Jitendra Malik
Adriana Romero
Michal Drozdzal
Venue: NeurIPS 2021 Poster (2021)
Keywords: active touch
active sensing
3D reconstruction
3D perception
robotics

Abstract: Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. In active touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limited by the lack of frameworks for shape exploration. In this paper, we focus on this problem and introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration. Our framework enables the development of the first fully data-driven solutions to active touch on top of learned models for object understanding. Our experiments show the benefits of such solutions in the task of 3D shape understanding where our models consistently outperform natural baselines. We provide our framework as a tool to foster future research in this direction.

Active 3D Shape Reconstruction from Vision and Touch
Edward Smith 1.2+
David Meger?
Luis Pineda'
Roberto Calandra'
Jitendra Malik
Adriana Romero-Soriano 1.2.
Michal Drozdzal 1,t
Facebook Al Research
McGill University
University of California, Berkeley
Abstract
Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied static datasets of limited sensory data such RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. In active touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limited by the lack of frameworks for shape exploration. In this paper, we focus on this problem and introduce system composed of: a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh- based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration. Our framework enables the development of the first fully data- driven solutions to active touch on top of learned models for object understanding. Our experiments show the benefits of such solutions in the task of 3D shape understanding where our models consistently outperform natural baselines. We provide our framework as tool to foster future research this direction.
Introduction
3D shape understanding is an active area of research, whose goal is to build 3D models of objects and environments from limited sensory data. It is commonly tackled by leveraging partial observations
multiple view RGB images [16] 25 31. depth Most of this research focuses building shape reconstruction models from a fixed set of partial observations. However, this constraint is relaxed in the active sensing scenario, where additional observation can be acquired to improve the quality of the 3D reconstructions. In active vision 4,, for instance, the objective can be to iteratively select camera perspectives from an object that result in the highest improvement in quality of the reconstruction (76) and only very recently the research community has started to leverage large scale datasets to learn exploration strategies that generalize to unseen objects (77
such as single view RGB image 660 67
44
maps [61 74 tactile readings 7 49 59
45
38
51 29 78 50 6).
Human haptic exploration of objects both with and without the presence vision has classically been analysed from a psychological perspective, where it was discovered that the developed tactile exploration strategies for object understanding were demonstrated to not only be ubiquitous, but also highly tailored to specific tasks [37 33. In spite of this, deep learning-based data-driven approaches to active touch for shape understanding are practically non-existent. Previous haptic exploration works consider objects
*Correspondence ejsmith @ fb.com and edward @ mail.megill.ca 'Equal Contribution
35th Conference on Neural Information Processing Systems PPe 2021), virtual.

---

Poliey
Legend
Simulator
Signal
Reconstruction Natwork
All Settings
Target
Predicted Shape
Chanfer
Extracted Touch
Signals
Figure An overview our active touch exploration framework. Given 3D model, the simulator extracts touch and vision signals that are fed to the reconstruction model. The reconstruction model predicts 3D shape that is used as an input policy model that decides where touch next. The policies are trained to select grasps which minimize the Chamfer Distance.
independently, and build uncertainty estimates over point clouds, produced by densely touching the objects surface with point-based touch sensors 7 73 28 46). These methods do not make use of learned object priors, and large number of touches sampled an object's surface (over 100) is necessary to produce not only a prediction of the surface but also to drive exploration. However, accurate estimates object shape have also been successfully produced with orders of magnitude fewer touch signals, by making of high resolution tactile sensors such as [75], large datasets of static 3D shape data, and deep learning see e.g. [56, 68 Note that no prior work exists to learn touch exploration by leveraging large scale datasets. Moreover, no prior work explores active touch the presence of visual inputs either (e.g. an RGB camera).
Combining the recent emergence of both data-driven recon- struction models from vision and touch systems [56 688, and data-driven active vision approaches, we present novel formu- lation for active touch exploration. Our formulation designed to casily enable the of vision signals to guide the touch exploration. First, we define a new problem setting over active touch for 3D shape reconstruction where touch exploration strategies can be learned over shape predictions from learned reconstruction model with strong object priors. Second, develop a simulator which allows for fast and realistic grasping of objects, and for extracting both vision and touch signals using robotic hand augmented with high-resolution tactile sensors [75]. Third, we present 3D reconstruction model from vision and touch which produces mesh-based predictions and achieves impressive performance in the single view image setting, both with and without the presence of touch signals. Fourth, we combine the simulator and reconstruction model to produce a tactile active sensing environment for training and evaluating touch exploration policies. The outline for this environment can be viewed in Figure] []
(a) Touch only exploration.
(b) Touch exploration with visual prior.
Figure 2: Target objects (top rows) and predicted 3D shapes (bottom rows) after grasps have been selected.
Over the provided environment, present series of data- driven touch exploration models that take as an input mesh-
based shape reconstruction and decide the position of the touch. By leveraging large scale dataset of over 25k CAD models from the ABC dataset (34] together with our environment, the data-driven touch exploration models outperform baseline policies which fail to leverage learned patterns between object shape or the distribution of object shapes and optimal actions. We demonstrate our proposed data-driven solutions perform up 18 % better than random baselines and lead to impressive object reconstructions relative to their input modality, such as those demonstrated in Figure 2 Our framework, training and evaluation setup, and trained models are publicly available on a GitHub repository to ensure and encourage reproducible experimental comparison
http:///gthhooccm///sssoooesstse and Touch

---

2 Related Work
3D reconstruction. 3D shape reconstruction is a popular and well studied area of research with a multitude of approaches proposed across varying 3D representations [13 63
19 52
67 23 12 7T
and input modalities [27 53 69 57]. While still a niche topic, a number of works have attempted 38). These works all assume point based touch signals which supply at most one point of contact of normal information. A limited number of methods have also been proposed for 3D reconstruction from both vision and touch signals 26 22 68
to reconstruct 3D shape from only touch signals 7 49 59
45
69 56] In contrast to these works which use static datasets, we focus on improving 3D reconstruction through tactile exploration. Shape from interaction methods have also been proposed for object reconstruction through hand interactions
42 64 47 though in contrast to our work an image of the interaction provides the additional information rather than touch readings.
Active sensing for reconstruction. Active vision aims to manipulate the viewpoint of a camera in order to maximize the information for particular task M 76.. Similar to our setting, active vision can be useful in 3D shape reconstruction 62 70, 17
35| but where camera perspectives are planned instead of grasp locations. Only very recently have deep learning active vision approaches been proposed for 3D object reconstruction [77 41 51]. Deep learning based active sensing for reconstruction has also recently emerged in the medical imaging domain, where the time spent performing MRI scans has been reduced by learning to select a small number of more informative frequencies over a pre-trained reconstruction model
41
14
29
50 6
Touch-based active exploration for shape understanding. Initial works tackling active tactile explo- ration focused object recognition. [1H3]. While some of these methods do integrate both vision and touch for active shape understanding, they only focus on object recognition, operate over small datasets of already known and observed objects and do not leverage deep learning tools for learning patterns over large datasets of object shapes. Fleer and Moringen et. al. proposed to learn haptic exploration strategies using a reinforcement learning framework over recurrent attention model, however these strategies were optimized for object classification and only trained over a dataset of 4 objects with a single, floating, depth-based tactile sensor array (20)
Several priors works have explored the problem of active acquisition of touch signals specifically for for surface reconstruction. A common theme among most approaches is to predict object shape using Gaussian processes so that uncertainty estimates are naturally available to drive the selection of the next point to touch, which is performed using single finger [28 73 18. To improve the speed of shape estimation during tactile exploration Matsubara et. al. [40] consider both uncertainty and travel cost when selecting touches using graph-based path planning. Bierbaum et al. [8] instead drove exploration through a dynamic potential field approach made popular in robot navigation to produce point cloud predictions, and extracted touch signals using a comparatively unrealistic 5 finger robotic hand model in simulation. These methods all make use of point based tactile sensors and use deterministic strategies, which are tuned and evaluated over very small number of objects. In contrast to these works, our approach is fully data driven, and as such, actions are selected using policies trained and evaluated over tens of thousands of objects. Moreover, predictions are made in mesh space fusing visual information with high resolution tactile signals extracted from objects in simulation using a realistic robot hand.
3 Active touch exploration
In our proposed active touch exploration problem, given a pre-trained shape reconstruction model over touch and optionally vision signals, the objective is to select the sequence of touch inputs which lead to the highest improvement in reconstruction accuracy. To tackle this problem, we define an active touch environment that contains simulator, reconstruction model (a pre-trained neural network), and loss function. The simulator takes as input a 3D object shape O together with parameters describing grasp, g. and outputs touch readings, t, of the 3D shape at the grasp locations along with an RGB image of the object, 1. The reconstruction model is neural network parametrized by o, which takes an input X and produces the current 3D shape estimate Ô as follows: Ô = f(X; ø). In our setup, we investigate two reconstruction model variants which differ in their inputs: 1) the model only receives a set of touch readings, 1, such that X = I, and 2) the model receives both set of touch readings, 4, and an RGB image rendering of the shape, 7, such that X = T). The loss function takes as input the current belief of the object's shape, Ó, and the ground truth shape, O, and computes the distance between them: d(O, Ô). Thus, active touch exploration can be formulated as sequentially selecting the optimal set of K grasp parameters (0), 92. gK) ) that maximize the similarity between the ground truth shape O and the
3

---

Lagard Vilion Truch Vision Sagnal
ABl Settingg
Loodoogee
Vnagine posullle
Sed honn selecced actien
Mive tanditt saffaze
Perkor grinp
adiora
Figure 3: Steps used in the tactile grasping simulator to produce simulated vision and touch signals.
reconstruction output after K grasps Öx
arg min ((0,OR).
where She determine the touch readings fed to the reconstruction network producing Óz. We use the Chamfer distance (CD) [60] between the predicted and target surface as the distance metric in our active touch formulation. Further details with respect to the CD are provided in the supplemental materials. In the reminder of this section, we provide details about the touch simulator and the reconstruction model.
3.1 Active touch simulator
Conceptually, our simulator can be described in the five steps depicted in Figure First, an object is loaded onto the environment. Second, an action space is defined around the 3D object by uniformly placing 50 points on a sphere centered at the center of the loaded object. Third, to choose a grasp, one of the points is selected and a 4-digit robot hand is placed such that its 3rd digit lies on the point and the hand's palm lies tangent to the sphere. Fourth, the hand is moved towards the center of the object until it comes in contact with it. Last, the fingers of the hand or are halted by the contact with the object. As a result, the simulator produces 4 touch readings(one from each finger of the hand) and one RGB image of the object. Note that each action is defined by its position index on the sphere of 50 actions. This paramcterization is selected specifically as it does not require any prior knowledge of the object, other than its center, and in simulation it consistently leads to successful interactions between the hand's 's touch sensors and the object's surface.
closed until they reach the maximum joint angle,
In simulator, all steps performed in python across the robotics simulator PyBullet [15] the rendering tool Pyrender [39], and PyTorch [48]. For a given grasp and object, the object is loaded into PyBullet [15,, along with Wonik's Allegro Hand [54] equipped with vision-based touch sensors (36) each of its fingers, and then the point in space corresponding to the action to be performed is selected and the grasping procedure is performed using PyBullet's physics simulator. Pose information from the produced grasps is then extracted and used by Pyrender to render both a depth map of the object from the perspective of each sensor and an RGB image of the object from fixed perspective. The depth maps are then converted into simulated touch signals using the method described in [56]. All steps in this procedure are performed in parallel using GPU accelerated computing, and as a result across the 50 grasping options of 100 randomly chosen objects, simulated grasps and touch signals are produced in ~ 0.0317 seconds each on Tesla V100 GPÚ with 16 CPU cores. Our simulator supports two modes of tactile exploration grasping and poking. In the grasping scenario, the hand is performing a full grasp of an object using all four fingers. While, in the poking scenario, only the index finger of the hand is used for touch sensing. Further details on this simulated environment are provided the supplemental materials.
3.2 Shape reconstruction
We take a chart-based approach to reconstruction (24), beginning from (56), which used charts to fuse vision and touch signals for shape prediction, and extending it to effectively leverage touch positional information while handling increasing number of touches, and to efficiently predict the object shape from the touch readings In partieular, shape is predicted in the mesh representation by repeatedly deforming a collection of independent mesh surface elements - i.e. charts using a graph convolutional network (GCN). The full pipeline for this reconstruction method is highlighted in Figure
Predicting local shape from touch readings. Available touch signals are passed through a touch convolutional neural network (CNN), which takes set of touch readings as input, and produces set of mesh surface elements, referred to as touch charts, representing the surface of the touched object where the touches occurred. We train the touch CNN to directly minimize the CD (60] between the predicted touch charts and the local surface at the corresponding touch site. By contrast, [56) predicts local point clouds which are then converted to charts via iterative optimization.
We use subscript K to indicate that the reconstruction has been obtained after observing K touch readings.

---

Obpeci Prior
Repeat Deformaion
Viloon CNN
Meah
Charl
Meshh
Chart
Vidoo 5 goal
Foutures
Predicted Shape
Legend
Shope
Vieon and Truchh
Toucch ChN
H6
Einbacdingg
Touch
Figure 4: Our pipeline to 3D object reconstruction from vision and touch.
Chart-based mesh representation and vertex features. A mesh in the shape of a sphere is initially created from a large collection of mesh surface elements, known as charts, where each vertex is defined by its location. We call these mesh surface elements vísion charts in the presence of vision signals, and touch charts in their absence. When leveraging touch signals, the mesh of initial charts is augmented with set of N additional touch charts, which may be initialized with the previous touch CNN vertex location predictions or uninitialized i.e. all vertex locations are set to zero - where N denotes the maximum number of touch charts expected in the active exploration process. Unlike (56). encode the vertex locations by leveraging positional embeddings [43 55 [66] to capture higher frequency shape information. When required, mask embedding appended each vertex in the mesh indicate if it originates from an uninitialized touch chart, a predicted touch chart, or a vision chart. With this setup, a variable number of touch charts can be expected by the subsequent deformation network. In addition, if vision signal is available, the image is passed through standard CNN and the extracted vision features are projected onto the vertices of the mesh using perceptual feature pooling 667 58 56). Thus, the resulting mesh is composed of touch and eventually vision charts, whose vertex features include the
positional vertex embedding, mask embedding when needed, and if available, the corresponding projected visual features. The mesh connectivity enabling communication among vertices and charts is defined following [56].
Mesh deformation process. The chart-based mesh representation is fed to a mesh deformation model composed of two GCNs, by contrast (56) used single GCN to handle the mesh deformation process. Decoupling the iterative GCN of [56] is crucial to ensure the use of vertex positional information by the model. Since the mesh deformation process of any object starts with the same sphere, parameter sharing across iterations results in the model ignoring the positional vertex information in the predictions of the initial shape belief, hindering the use of the added touch charts in subsequent iterations. In our model, the first GCN aims to learn an object prior, which can be learnt either from vision signals (vision prior) or from touch signals (touch prior). Note that the expected input of the first GCN is static w.r. the position of vision and uninitialized touch charts. The second GCN takes the object belief resulting from the first GCN and refines through 2-step deformation process, in which each step recomputes the projection of the image features onto the mesh. Since the input to the second GCN is expected to be different in each case, we can leverage parameter sharing in its 2-step deformation process. Note that touch signals are only included in the second GCN when leveraging vision signals; however, they are included in the first GCN in the touch-only setting when available.
Training. The parameters of the whole reconstruction pipeline, including both GCNs and the vision CNN, are jointly optimized to minimize the CD between the predicted and target surface 660 (88. With this setup, potential vision signal and a variable number of touch signals can be leveraged to produce surface prediction in a single model pass. Further details as well as a comparison highlighting the superiority of our shape reconstruction pipeline w.z. [56] can be found in the supplemental material.
4 Data-driven policies for touch exploration
Our touch exploration framework takes advantage of the reconstruction model introduced in Section 3.2 to predict 3D shapes in the mesh space, and defines policies to select the positions of the next touches to acquire in order to maximize the similarity between predicted and target shape. While useful for graphics applications and efficient for representing surfaces, meshes are difficult to process and computationally heavy to compare. To combat these issues, we propose to use mesh embeddings of reduced dimensionality to facilitate the learning of our policies. The mesh embedding extracted from the bottleneck of a mesh autocncoder which is trained offline from the shape predictions to produce learned embedding space.
5

---

We also use the mesh embeddings to allow for efficient distance metric computation over predicted shapes i.e. Euclidean distance in the embedding space.
Autoencoder for shape embedding. The encoder, e, takes as an input a surface mesh, and produces mesh embedding. Following our shape reconstruction model, we use positional embeddings 43 55 66 to represent the vertices in the mesh. The mesh is then passed through series Zero-Neighbor GCN layers [58 32 update the vertex features, followed by a channel-wise max pooling operation across vertices to produce a latent encoding. The decoder takes the resulting latent encoding and follows the FoldingNet 72 architecture to yield point cloud with 2,024 points recovering the object shape. The autoencoder is trained by minimizing the CD between the input mesh and the predicted point cloud.
In the remainder of this section, we outline the object-specific and dataset-specific policies selected for the purpose of comparison. In object specific policies the current object shape is considered when deciding which action to perform. In dataset-specifie policies the full training set of objects is leveraged to determine the optimal fixed trajectory to perform across all test set objects.
4.1 Object-specific policies
Nearest Neighbor (NN), We compute myopic greedy trajectories for all objects in the training set, and save the object reconstruction at each time step along with the action grasping parameters - leading to its best immediate improvement. Then, when evaluating, we search our record of training reconstructions and their corresponding actions to find the most similar reconstruction to the current object belief and simply copy the grasping parameters from our record.
To allow for easy and efficient comparisons, we leverage the learned mesh encoder, e, to perform similarity search with the lz distance.
Supervised (Sup.). In this model we attempt to learn the improvement each action will provide for given object using regression. An independent network h; is trained for each time step i to predict the relative improvement which will result from taking each action. Each network is comprised of set of fully connected layers with ReLU activations and takes input the set of already performed grasp parameters, and the embedding of the current and initial shape reconstruction produced by the pre-trained autoencoder. The networks trained sequentially, such that the i-th network learns to predict the relative improvement of each possible i-th action after performing the actions predicted by the networks of the previous time steps. When evaluating the performance of the supervised approach at time step h2 is used to determine which action will lead to highest improvement, and this action, Ji, is selected:
9. mrg h,(1go, .... 91-1), (l-).,.)--),e(00).e(()))))
where G represents the set of all grasps.
Double Deep Q-Networks (DDQN). In the last object-specific model we leverage the discrete deep RL method Double Deep Q-Networks (DDQN) (55. In our case, the value network takes as input the set of actions already performed and the embedded current reconstruction of the object, and predicts value for every possible action. We propose two value network architectures. In the first, referred as DDQNm. the value network takes as input the mesh reconstruction of an object, where an embedding of the performed actions appended to every vertex's 's feature vector. The network architecture identical to that of encoder, e, and produces small shape embedding, which is then passed through a few fully connected layers to predict a value for every action. In the second, referred to as DDQN, the current reconstruction is passed through the pre- pre-trained mesh encoder, e, producing a shape embedding which is concatenated with the embedding of the actions performed, and fed through a few fully connected layers to predict value for every action. Note that the first setup benefits from complete understanding of the current shape belief, as the mesh is fed to the value network. The second setup benefits from the pre-computed shape embeddings, which already contain the rich information necessary for reasoning over the object and allows for simplistic network design which has already been demonstrated to perform In both cases, the action selected is the one
well in deep reinforcement learning (RL) settings which the value network Q predicts has the highest value:
91
arg max
(3)
GV GV.

---

4.2 Dataset-specific policies
Most frequent best action (MFBA).
This policy selects the first action by computing the performance of all actions over all objects in the training set, and then chooses the most common best action. For the second action, the performance of performing the first fixed action followed by all remaining actions is computed, and the most common best performing second action is chosen. This is repeated until a full trajectory is obtained. Then, when evaluating, this trajectory is selected every time, regardless the object reconstruction.
Lowest error best action (LEBA). This policy is in effect identical to the MFBA except that the action which leads the greatest average error improvement selected and fixed at every time step rather then the most common best action.
5 Experiments
In this section, we validate the reconstruction and autoencoder models. We then compare object-specific and dataset-specific policies to several basclines. Additional experimental details can be found in the supplementary material.
5.1
Experimental setup
5.2
Shape reconstruction from static vision and touch signals
Dataset of CAD models. The dataset used is made up of 40,000 objects sampled from the ABC dataset [34] [56). CAD model dataset of approximately one million ob- jects. This dataset poses a much harder generalization challenge than other 3D object datasets, such as [00] due to its highly variable object shapes, and lack of clearly defined classes over which biases can be learned. The geometry of these objects were decimated such that all objects possess approximately 500 vertices. Those ob- jects which could not be reduced to this size due to ge- ometric constraints and those which possessed multiple disconnected parts were automatically removed, leading to set of 26,545 usable object models. This set of objects was split into 5 sets; 3 training sets cetssoof of size 7,700 object each, a validation set comprised 2,000 objects, and a test set of size 1,000.
Grasp #
Model
To [56]
25,586 =0.069
9.016 00.358
To [ours]
24.864 =02266
8.220 00.38
V&Ta [56
2.653 1:0.022
2.637
100042
V&Tg [ours]
2.538 1.0.98
2.486
10.102
Table 1: Comparison between our reconstruc- tion model and state-of-art in terms of CD.
Baselines and Oracle.
(1) Random baseline. As naive baseline, a random policy is considered, which selects for every time step and object one of the available actions uniformly at random. This is the standard baseline for any exploration algorithm. (2) Even baseline. As a second naive baseline, we consider policy which randomly selects an evenly spaced set of 5 actions over the sphere of possible actions. This baseline is chosen it will result in uniform coverage of the target object, which is intuitively useful and strong strategy for object understanding in our task (3) Oracle, As near-optimal target for the performance of our policies a myopic oracle policy is considered. In this policy, for given object and time step the action which resulted in the best improvement is selected.
This policy possesses unfair hindsight, which is not accessible to all others, and should be seen as an upper-bound point of comparison. The true optimal policy cannot be computed in a reasonable time frame, however due to the diminishing return of rewards from actions in this framework, the provided myopic oracle policy represents a close approximation.
Experimental scenarios. We examine the performance of our exploration framework and models across learning settings: (1) poking only, Tp. where only touch from the third finger of the hand is leveraged; (2) grasping only, Tg. where only touch signals from all hand sensors used during shape exploration: (3) poking with vision, V&Tp, an extension of Tp that includes a visual input signal; and (4) grasping with vision, V&Tg an extension of To that leverages visual signal.
We evaluate the performance of our proposed reconstruction method in the target domain of 3D recon- struction from vision and touch, and compare it to the current state-of-the-ard [56]. While Wang et. al.
'Three training sets are used to mitigate the progressive overtitting which might occur from sequentially training the elements of our pipeline (reconstruction, autoencoder and policy networks) on the same data.
7

---

Figure 5: Two objects from the test set, along with their four nearest neighbors the measured in the latent space of our trained autoencoder (V&TG setting).
Baselines
Dataset specific MFBA
Object-specific
Input
Oracle
Random
Even
LEBA
NN
DDQN
DDQN:
Sup
Tp
19.35
36.38
33.25
32.40
29.85
33.46
32.41
31.10
31.21
-0.00
+:029
+:048
+1.04
+0.39
#0.51
#0.40
+0.34
#0.67
To
16.38
25.83
24.53
23.46
23.04
24.34
23.92
23.84
23.70
0.00
1:0.14
1:0.27
::0.07
1:009
::0.29
2:144
+:0.3
10.27
V&Tp
78.95
94.56
93.95
93.59
92.36
91.79
93.75
92.62
93.12
=0.00
1:0.4
1:0.99
1:0.32
1.0.5
1:0.55
10.48
1..38
V&Ta
77.18
90.65
90.29
89.39
89.31
88.53
90.07
89.32
89.46
10.34
1.0.55
1:0.25
10.51
1.0.23
Table 2:
Comparison of active touch exploration strategies. Numbers represent a ratio between CD after actions and initial CD (lower is better).
68 do consider both vision and touch for reconstruction, touch is not fused directly for prediction but rather used for shape refinement in sim2real transfer. The results of this experiment can be seen in Table where the highest performing models have been selected from the validation set while doing hyper-parameter search, and mean and variance numbers across these 4 models the test set are shown. From these results is clear that the proposed method outperforms the baseline comparison in all settings, and validates its model choices in our target multi-modal domain.
5.3 Shape Autoencoder
For each of the four learning settings an autoencoder is trained by leveraging the output of the reconstruc- tion model. We qualitatively validate the shape embedding learnt by our autoencoder by visualizing object shapes and their nearest neighbors in the learnt embedding space. Figure 5 depicts 2 random objects sampled from the test set, along with the 4 other objects closest to their latent encoding in the test set for the V&TG setting. Moreover, in the supplemental materials we highlight the average CD between the input and output meshes across 5 grasps, our learning settings, and observe that they low relative to the error of the corresponding reconstruction models. The visual similarity of objects to their closest neighbors in the latent space along with the relatively low CD achieved demonstrates that the learned latent encodings possess important shape information which may be leveraged in the proposed active exploration policies.
5.4 Active Touch Exploration Results
We examine the performance of all described active touch exploration strategies over grasps and show the results all policies over the 4 learning settings in Table For the DDQN DDQN,, and Supervised policies, the 5 highest performing models have been selected from the validation set while doing hyper- parameter search, and mean and variance numbers across these models on the test set are reported. For the Random, and Even results, the strategies are repeated 5 times over the test set. For the MFBA, LEBA, and NN results, 5 random subsets 40% of the training data are made to produce different fixed trajectories, training set latent distributions. each strategy in the TG and V&TG settings. Here, the points of all actions on the sphere are transformed into their corresponding UV coordinates in image, and the intensity value for every pixel corresponding to an action is set to its relative frequency computed over the test set. The visible area of the sphere of actions from the camera's 's perspective is highlighted in orange, and non-visible in blue. Figure highlights object predictions after grasps with action selected using the DDQN; model in the TG and V&Ta settings.
the distributions of action selected by
From Table we can see the clear, and expected trend of the Oracle strategy being the best performing and the Random bascline being the worst, followed by the Even baseline which is slightly better. In all cases,

---

Oracle
Random
Even
MFBA
LEBA
NN
DDQN
DDQN:
Supervised
Tc
V&To
Figure 6: Distribution of selected actions (greyscale encoded) for all policies in the Ta and V&Ta settings, with visible area of the sphere of actions from the camera highlighted in orange.
Signal
Oracle
LEBA
NN
Sup.
Signal
Oracle
LEBA
NN
Sup.
Figure 7: Action selection for the Oracle, LEBA, NN, and Supervised strategies in the V&Tg setting, where the arrows indicate the direction the hand moves towards the object for each selected action.
LEBA performs better than MFBA, most probably due its selection strategy being directly in line with the target objective. In both touch only settings, the LEBA policy performs best. For the learned policies, even after 4 grasps with a full hand of sensors, enough information about the current shape must not be available to properly learn the best action perform. This is supported by the distributions in top line of Figure where object-specific polices seem to employ close to a fixed strategy. In contrast to LEBA, these polices cannot leverage the full training dataset simultaneously and so their fixed strategy performs worse. From this we can see that even in the absence of meaningful shape information, leveraging the dataset of objects provides significantly better action selections leading to improved reconstruction accuracy.
In the vision and touch settings (V&Tp and V&Tg), we see a reverse trend, with the NN policies performing best in both grasping settings. From this can that in the presence of better shape understanding (due to additional vision input), more successful action selection can be discriminated per object. This is supported in Wieee where see that with vision priors, far more variable actions performed, meaning that the object shape is now being appropriately considered. Moreover, in row 2 of F trend can be observed where some policies avoid actions where the touched surface more likely to be visible from the camera. In the V&TG setting, the NN. DDQNM, and supervised policies select visible actions only 40.20%, 45.91 % and 43.98% of the time respectively. Compared to the random policy which selects these actions 48.25 % of the time, and the oracle which selects them 43.80% of the time, this indicates that these policies have learned the intuitive strategy of avoiding sampling areas of the surface which have already been observed. In Figure 7 the different action selection strategies employed by various policies and the Oracle are shown for the V&TG over 2 randomly sampled objects in the test set. Here the green arrows indicated from which direction the hand moves towards the objects for all 5 action selected. In both cases we see the object specific strategies tend to be fairly uniform around the sides, possibly to correctly identify the unseen dimensions of the objects.
6 Discussion
Limitations. There exist some limitations which should be highlighted. First, the reconstruction method aims exclusively minimize the CD, which leads to poor visual object quality in the mesh representation 23, 677. While attractiveness regularizers are available, we opted to focus exclusively on accuracy to make clear distinctions improvement from information across different touch options. This is at the cost of visual quality. Second, the chosen shape agnostic grasp parameterization, where the hand always moves towards the center of the object leads touch sites biased towards the center of objects which possess dramatically different dimensional scales. An example of this can be seen the the first object of Figure 7 where because the object long and thin, all touches will lie in the center of the object, as highlighted by the direction of the arrows. Finally, our environment requires full 3D shape supervision for training, which while easily available in simulation, limits its application to real world scenarios.
Societal Impact. Our work contributes towards improved understanding of the three dimensional world in which we all live. In particular, we provide new framework to study touch perception in a simulated environments. Thus, we advance the understanding of the importance of haptic information in the task of active 3D understanding, especially when used in tandem with visual signals. We envision our contributions to be relevant for real world robot-object manipulation. However, the improved active 3D

---

object understanding could have positive impact in fields beyond robotics, such as automation and virtual reality. Failures of these models could arise if not enough testing is performed prior to the deployment of the automation tools. To mitigate these risks, we encourage further investigation focusing on active 3D reconstruction system generalization limits both in the simulated and real-world scenarios.
Conclusions. In this paper, we explored the problem of data-driven active touch for 3D object recon- struction from vision and touch. We introduced tactile-grasping simulator which allows for the efficient production of vision and touch signals from selected grasp parameters, and built a new 3D reconstruction method from vision and touch which achieves impressive performance both with and without haptic inputs. Over these elements and a large dataset of simulated objects, we built an active touch exploration environment which allows for the training and testing of active touch policies for 3D shape reconstruction. Finally, we produced an array of data-driven active touch policies which we compared to set of baselines. The benefit of leveraging data for active touch then highlighted by the superior reconstruction results of learned policies both in the presence of poor and rich shape information. In the presence of only touch information, the most successful exploration strategies learn deterministic trajectory over the training data to employ indiscriminately over test objects even in the presence of shape information, indicating that either not enough information is present or that this information cannot be learned over with the current models. In the vision and touch settings the most fruitful strategies learn to select grasps based on the current belief of the objects' shape, and experiments also indicate that learned models tend to favour grasps which occur occluded areas of the objects' surface.
10

---

References
[1] Peter Allen. Surface descriptions from vision and touch. In JEEE International Conference on Robotics and Automation (ICRA), volume I, pages 394-397. IEEE, 1984.
[2] Peter K Allen. Integrating vision and touch for object recognition tasks. The International Journal of Robotics Research, 7(6):15-33, 1988.
[3] Peter K Allen and Paul Michelman. Acquisition and interpretation of 3-d sensor data from touch. 1990.
[4] Y. Aloimonos, I. Weiss, and A. Bandyopadhyay. Active vision. International Journal Computer Vision, 1:333-356, 1988.
[5] Kumar Ashutosh, Saurabh Kumar, and Subhasis Chaudhuri. 3d-nvs: A 3d supervision approach for next view selection. arXiv preprint arXiv:2012.01743, 2020.
[6] Tim Bakker, Herke van Hoof, and Max Welling. Experimental design for mri by greedy pol- icy search. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18954-18966. Cur- ran Associates, Inc., 2020. URL -appdd pdf
wwwerdiiinnnnnnunnnneeeeeeeeeeee neurips wwtpeiis.ccpppppeeeeee cc/paaper/00000//iiee
[7] A. Bierbaum, Gubarev, and R. Dillmann. Robust shape recovery for sparse contact location and normal data from haptic exploration. In 2008 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3200-3205, 2008
[8] Alexander Bierbaum, Matthias Rambow. Tamim Asfour, and Rudiger Dillmann. A potential field approach to dexterous tactile exploration unknown objects. In TEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 360-366. IEEE, 2008.
[9] Marten Björkman, Yasemin Bekiroglu, Virgile Högman, and Danica Kragic. Enhancing visual perception of shape through tactile glances In IEEE International Conference on Intelligent Robots and Systems (IROS), 11 2013. doi: 10..1///...2021366666666
[10] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
[11] Shengyong Chen, Youfu Li, and Ngai Ming Kwok. Active vision in robotic systems: A survey of recent developments. The International Journal of Robotics Research, 30 (3311):1333-137) 2011.
[12] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6970-6981, 2020.
[13] Christopher B Choy, Danfei Xu, Jun Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 628-644. Springer, 2016
[14] CI Connolly, The determination of next best views. In Proceedings. 1985 IEEE international conference on robotics and automation, volume 2, pages 432-435. IEEE, 1985.
[15] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. GitHub repository, 2016.
[16] A. Dame, V. A. Prisacariu, C. Y. Ren, and I. Reid. Dense reconstruction using 3d object shape priors. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1288-1299 2013.
[17] Jeffrey Delmerico, Stefan Isler, Reza Sabzevari, and Davide Scaramuzza. A comparison of volumet- ric information gain metrics for active 3d object reconstruction. Autonomous Robots, 42(2):197-208. 2018.
11

---

[18] Danny Driess, Peter Englert, and Mare Toussaint. Active learning with query paths for tactile object shape exploration. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
[19] Haoqiang Fan, Hao Su, and Leonidas Guibas. A point set generation network for 3d object reconstruction from a single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 38, 2017.
[20] Sascha Fleer, Alexandra Moringen, Roberta L Klatzky, and Helge Ritter. Learning efficient haptic shape exploration with rigid tactile sensor array. PloS one, 15()):e0226880, 2020.
[21] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[22] Gabriela Zarzar Gandler, Carl Henrik Ek, Märten Björkman, Rustam Stolkin, and Yasemin Bekiroglu. Object shape estimation and modeling, based on sparse gaussian process implicit surfaces, combining visual data and tactile exploration. Robotics and Autonomous Systems, 126:103433, 2020. ISSN 0921-8890. doi: https://60..rr//000011///
101 10to 2020.103333..... htthttp:// sciencedirectt
com/ htte:/sceenceeeeee httpa////rrrrcce//////1/11111110
[23] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. IEEE International Conference on Computer Vision (ICCV), 2019.
[24] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 216-224, 2018.
[25] C. Häne, N. Savinov, and M. Pollefeys. Class specific 3d object shape priors using surface normals. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652-659, 2014.
[26] Jarmo Ilonen, Jeannette Bohg, and Ville Kyrki. Three-dimensional object reconstruction of symmet- ric objects by fusing visual and tactile sensing. The International Journal of Robotics Research, 33 (2):321 12):321-341, 2014.
[27] Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervised learning of shape and pose with differen- tiable point clouds. arXiv preprint arXiv: arXiv:1810.0938// 2018.
[28] N. Jamali, C. Ciliberto, L. Rosasco, and L. Natale. Active perception: Building objects' models using tactile exploration. In JEEE- -AS International Conference on Humanoid Robots (Humanoids), pages 179-185, Nov 2016. doi: 10.1 200HUUMMANIDDSS216..7080000
[29] Kyong Hwan Jin, Michael Unser, and Kwang Moo Yi. Self-supervised deep active accelerated mri. arXiv preprint arXiv: 1900194497.477 2019.
[30] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo ma- chine. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS). pages 365-376. Curran Associates, Inc., 2017. URL. http::/papers: nips htaps.cc//aae cc/paper/ (640--Lerrnnggaamillee a- www.mii--ie---teee-e-eeee
[31] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, and Peter Henry. End-to-end learning of geometry and context for deep stereo regression. In IEEE International Conference on Computer Vision (ICCV), pages 66-75 IEEE Computer Society, 2017. doi: 10. 10.1 10.110///122. URL.
https://doi.. org/ wtt.////.....///111111
.0177.111
[32] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
In Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017. URI. nttps://openreviee .nw. wtwpr/ewwnneeffuuiiiiieeeee forum??dd-SJJAAayYy
[33] Roberta L. Klatzky, Susan Lederman, and Dana E. Matula. Haptic exploration in the presence of vision. Journal of Experimental Psychology: Human Perception and Performance, 19(4):726, 1993.
12

---

[34] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Mare Alexa, Denis Zorin, and Daniele Panozzo. Abc: big cad model dataset for geometric deep learning. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[35]
Adarsh Kowdle, Yao-Jen Chang, Andrew Gallagher, and Tsuhan Chen, Active learning for piecewise planar 3d reconstruction. In CVPR 2011, pages 229-936. IEEE, 2011.
[36] Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang. Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, and Roberto Calandra. DIGIT: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters (RA-L), 5(3):3838- 3845, 2020. doi: 10. 109/1 202......2072...555
[37] Susan Lederman and Roberta Klatzky. Extracting object properties through haptic exploration. Acta psychologica, 84(1):29 1993.
[38] Shan Luo, Joao Bimbo, Ravinder Dahiya, and Hongbin Liu. Robotic Tactile Perception of Object Properties: A Review. arXiv e-prints, art. arXiv:1711.0308 003810, November 2017.
[39] Matthew Matl. Pyrender. httpes///gthh...c.ocmmmaaa/////p 2019.
[40] Takamitsu Matsubara and Kotaro Shibata. Active tactile exploration with uncertainty and travel cost for fast shape estimation of unknown objects. Robotics and Autonomous Systems, 91:314-326, 2017.
[41] Miguel Mendoza, Irving Vasquez- Gomez, Hind Taud, L Enrique Sucar, and Carolina Reta. Supervised learning of the next-best-view for 3d object reconstruction. Pattern Recognition Letters, 133:224-233 2311, 2020.
[42] Damien Michel, Xenophon Zabulis, and Antonis Argyros. Shape from interaction. Muchine Vision and Applications, 25(4):1077-1087, 2014.
[43] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[44] J Krishna Murthy, GV Sai Krishna, Falak Chhaya, and K Madhava Krishna. Reconstructing vehicles from single image: Shape priors for road scene understanding. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 724-731, IEEE, 2017.
[45] Simon Ottenhaus, Martin Miller, David Schiebener, Nikolaus Vahrenkamp, and Tamim Asfour. Local implicit surface estimation for haptic exploration. In IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 850-856, 11 2016. doi: 10. H0.100//MMNOO 2016. 7803372.
[46] Simon Ottenhaus, Lukas Kaul, Nikolaus Vahrenkamp, and Tamim Asfour. Active tactile exploration based on cost-aware information gain maximization. International Journal of Humanoid Robotics, 15(01):1850015 2018
[47] Paschalis Panteleris, Nikolaos Kyriazis, and AA Argyros. Recovering 3d models of manipulated objects through 3d tracking of hand-object interaction. In IEEE International Conference Computer Vision Workshops (OUI 2015-ICCVW 2015), Santiago, Chile, 2015.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8025-8035. Curran Associates, Inc., 2019. URL http::::/ppppp...eerrrreeeee www.eeessssrrrrr....ommm. ccppapr////p/pc//cccoocceeeeeepp 9015-pytorch- an- http:p/oorrrraaaaaamaaaaveeeeeee www.iiippettetttyeeeeeee
peae Liibrng--a.... library.
pdf
13

---

[49] Z. Pezzementi, C. Reyda, and G. D. Hager. Object mapping, recognition, and localization from tactile geometry. In 2011 IEEE International Conference on Robotics and Automation, pages 5942-5948, 2011.
[50] Luis Pineda, Sumana Basu, Adriana Romero, Roberto Calandra, and Michal Drozdzal. Active mr k-space sampling with reinforcement learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 23-33. Springer, 2020.
[51] SG Potapova, AV Artemov, SV Sviridov, DA Musatkina, DN Zorin, and EV Burnaev. Next best view planning via reinforcement learning for scanning of arbitrary 3d shapes. Journal of Communications Technology and Electronics, 65(11):1484-144
2020.
[52] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1(2):,, 2017.
[53] Jason Rock, Tanmay Gupta, Justin Thorsen, Jun Gwak, Daeyun Shin, and Derek Hoiem. Completing 3d object shape from one depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2484-2493 2015.
[54] SimLab. Allegro hand overview, 2016. URL http://wiki. wwt.kkiooooi..oooiiic........ com/
hlllgrooHandWkkiinndde /indeixiiiiiiix.ieeeie http/Allegro_Haaee Hand Overview [Online; accessed 25-May-20200
[55] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurlPS, 2020.
[56] Edward Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal. 3d shape reconstruction from vision and touch. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. volume 33, pages 14193 14206. Curran Associates, Inc., 2020. URL https::/pproceeddngg .df
heurips.
[57] Edward Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning (CoRL), pages 87-96, 2017.
[58] Edward J. Smith, Scott Fujimoto, Adriana Romero, and David Meger. Geometrics: Exploiting geometric structure for graph-encoded objects. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of P Proceedings of Machine Learning Research, pages 5866-5876. PMLR, 2019. URL http:///proceedingg.. mlr. www.prreaasaae v97 smi tha.i.hhh........ html
[59] Nicolas Sommer, Miao Li, and Aude Billard. Bimanual compliant tactile exploration for grasping unknown objects. Proceedings IEEE International Conference on Robotics and Automation, pages 6400-6407, 09 2014. doi:
[60] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B. Tenenbaum, and William T. Freeman. Pix3d: Dataset and methods for single- -mage 3d shape modeling. CoRR, abs/1804.04611 2018. URL //arxiv.// org/ aaaaaaas///8 1804. 04610
[61] Julián Tachella, Yoann Altmann, Nicolas Mellado, Aongus McCarthy, Rachael Tobin, Gerald S Buller, Jean-Y Yves Tourneret, and Stephen McLaughlin. Real-time 3d reconstruction from single- photon lidar data using plug-and-pla point cloud denoisers. Nature communications, 10(1):1-6, 2019.
[62] Glenn H Tarbox and Susan N Gottschlich. Planning for complete sensor coverage in inspection. Computer Vision and Image Understanding, 61(1)84-111, 1995.
[63] Shubham Tulsiani, Tinghui Zhou, Alexei Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 209-217. IEBE, 2017.
[64] Dimitrios Tzionas and Juergen Gall. 3d object reconstruction from hand-object interactions. In Proceedings of the IEEE International Conference on Computer Vision, pages 729-737, 2015.
14

---

[65] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[67] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. arXiv preprint arXiv: /04v01800..0164, 2018.
[68] S. Wang. J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 3d shape perception from monocular vision, touch, and shape priors. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1606-1613, Oct 2018. doi: 10.1 2011 8593430.
[69] David Watkins- Valls, Jacob Varley, and Peter Allen. Multi-modal geometric learning for grasping and manipulation. In 2019 International Conference on Robotics and Automation (ICRA), pages 7339-7345. IEEE, 2019.
[70] Chenming Wu, Rui Zeng. Jia Pan, Charlic CL Wang, and Yong-Jin Liu. Plant phenotyping by deep- learning-based planner for multi-robots. IEEE Robotics and Automation Letters, 4(4):3113-3120, 2019.
[71] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems 32, pages 492-502. Curran Associates, Inc., 2019.
[72] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 206-215, 2018.
[73] Zhengkun Yi, Roberto Calandra, Filipe Fernandes Veiga, Herke van Hoof, Tucker Hermans, Yilei Zhang, and Jan Peters. Active tactile object exploration with Gaussian processes. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10.1 11199/ROS.2016.7599177
2016. doi:
[74] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference on 3D Vision (3DV), pages 728-737, 2018.
[75] Wenzhen Yuan, Siyuan Dong, and Edward Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):7762, Nov 2017. ISSN 1424-8220. doi: 10.3390/s17122762. URL http://d http::/xxx.dii.ogg 10. /390/
[76] Rui Zeng, Yuhui Wen, Wang Zhao, and Yong-Jin Liu. View planning in robot active vision: A survey of systems, algorithms, and applications. Computational Visual Media, pages 1-21, 2020.
[77] Rui Zeng, Wang Zhao, and Yong-Jin Liu. Pe-nbv: A point cloud based deep network for efficient next best view planning. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 7050-7057. IEEE, 2020.
[78] Zizhao Zhang, Adriana Romero, Matthew J Muckley, Pascal Vincent, Lin Yang, and Michal Drozdzal. Reducing uncertainty in undersampled mri reconstruction with active acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2058,2058, 2019.
15


Reviews:
[{"review_id": "zoZzaoPsTpS", "reviewer": "Reviewer_Ksky", "summary": "The paper presents data-driven exploration strategies for 3D reconstruction using visual and tactile data, a haptic simulator and a mesh-based 3D shape reconstruction model.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an interesting problem. However it can be improved considering the following points. Some of the listed previous works also focus on improving 3D reconstruction through tactile exploration, e.g. choosing where to touch next without exhaustive exploration also demonstrating experiments using real data. Please clarify and highlight the differences of your work in comparison. Another related work that can be included: Active tactile exploration with uncertainty and travel cost for fast shape estimation of unknown objects. The experiments are only conducted in simulation, how would the results be affected in terms of noise and uncertainty which would be increased in case of real data. The difference in performace in Table 1 looks quite small. Why is [48] chosen as a baseline among all related works, please motivate the choice, also motivate the choice of baselines for the exploration experiments. Could you report the value of K (number of touches/grasps) in experiments? Could you summarise the main findings from the experiments based on all the exploration strategies?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "yH1DbMpv5Cs", "reviewer": "Reviewer_Dd2Y", "summary": "This paper presents an approach for active 3D reconstruction of objects based solely on touch, and optionally together with vision. The main idea is active perception -- estimate where to touch/see the object in order to better reconstruct it. The proposed method to solve this involves a neural network that fuses touch and vision signals to estimate 3D shape, and a policy that predicts where to touch next for best reconstruction performance. Data comes from a simulator that produces both touch and image signals (from the ABC dataset).", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would like to begin by listing several positives with this paper:\n\n- It addresses an important problem (3D reconstruction) and tries to do so using a new active perception paradigm.\n- The paper is well written and motivated.\n- The experiments are extensive and convincing.\n\nI like the ideas in the paper and the proposed approach. In the following, I will focus on comments, questions, and suggestions for improvement.\n\n- It would be beneficial to differentiate between related previous work [48,59] and the proposed work. The main difference is active perception, but this can be made clearer.\n\n- Figure 2 is missing some labels. What are the top and bottom rows?\n\n- Regarding related work, there is some existing literature on the idea of \"Shape from interaction\" which would have been nice to discuss. In general, the paper could do a better job of crediting previous papers that present similar ideas. Here's a start:\n\nShape from interaction, Michel, Zabulis, Argyros, 2014\n\nThe paper uses charts for reconstruction but does not cite the most relevant paper:\n\nA papier-mâché approach to learning 3d surface generation, Groueix et al. 2018\n\n- Line 133 mentions 4 touch readings, but it isn't immediately obvious why it's 4 (mention 4 fingers of the robot).\n\n- The simulator sounds great. Will this also be released publicly?\n\n- Is the figure 4 legend correct? Red is denoted as vision and touch but takes only vision signals in the top part.\n\n- I generally like the proposed approach and do not have too many complaints.\n\n- Regarding results and experiments:\n\n    - The results in table 1 show only a marginal improvement over [48], could the authors comment on why?\n    - I would have liked to see results on real data, but I understand this is a hard problem.\n\nOverall, this is a great paper that I would like to see published. If the authors can propose a plan to fix the issues above in the rebuttal, that would be great.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciated the paper allocating significant space to discuss limitations and broader impacts. Not many of the papers I have seen do this.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OwepdVCO9Lg", "reviewer": "Reviewer_1dhq", "summary": "This paper introduces and explores the task of active 3D reconstruction using visual and haptic data. Instead of relying on static data to perform reconstruction, the learning process is guided by exploration, from which new data can be accounted for in the reconstruction process by the proposed model. In order to enable this setting/task, the authors develop a simulator that provides both visual and touch signals that are fed to the model, and an environment for training and evaluating policies.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors introduce the task of active 3D reconstruction, which presents a different setting for reconstruction by leveraging exploration rather than static data, and develop a simulator and environment to enable the task.\n- The authors extend the previously proposed chart-based reconstruction method to their setting to better leverage positional information, as well as incorporate additional information collected from exploration.\n- The authors use 4 learning settings across a number of data-driven policies to evaluate their approach, including an oracle that represents an upper bound on the performance of the method. Experiments demonstrate the usefulness of active touch through the improved performance on 3D shape reconstruction of the learned policies.\n- The paper is well-written, especially when providing an overview of all the policies and baselines used in the experiments.\n\nWeaknesses:\n- There are a few design decisions that were skimmed over (e.g., How was 50 selected to be the size of the action space? How was N (max # expected touch charts) determined?)\n- The authors use mesh representations, in contrast to point cloud representations used by Smith et al. [48], which introduces additional parameterization with needing to learn an autoencoder to reduce the dimensionality of the features used in learning the policies. Since the autoencoder's objective is to minimize the CD between an input mesh and output point cloud, did the authors try directly using point clouds in their pipeline instead of meshes (i.e., the main difference from [48] being the composition of 2 GCNs for the mesh deformation model)?\n- While Table 1 demonstrates better reconstruction in terms of CD, the variance is noticeably higher. Is there any intuition for why this may occur?\n\n----------\n\nThank you to the authors for addressing and clarifying the concerns raised. My rating remains unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations and societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "0fKmiNbgsDw", "reviewer": "Reviewer_VhZy", "summary": "This paper presents a novel active search on touch parameters to improve the 3D shape reconstruction with single (touch) or dual (touch & vision) modalities. The proposed reconstruction network and the active policy show good performance on simulated objects.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tOn the method originality\nThis paper mainly has two contributions which are the modified reconstruction network based on [48], and the active search policy of grasp parameters. The whole design is reasonable and novel to me. However, since the modification on the network and the policies are a little direct and intuitive to me, which makes the technical contribution a little limited.\n2.\tOn the experiments\nMy major concern on the experiments is that they are all experimented in simulation. Since the object is easy to find or 3D printed in the real world, the robot hand (Allegro) is not that expensive (compared to Shadow robot hand), the tactile sensor is open-source for manufacturing with low-cost. And after [48] which has already put forward the discussion of vision-touch learning, I think at this moment, a real-world experiment is needed. I wonder if the simple policy will also work in the real world, the evident challenges would be natural occlusion of object shape, and the displacement after touching.\n\nIf the training requires fairly large amount of data, how about apply some simple sim2real strategy? As touch may involves many contact-rich processes, I am suspicious of how well the simulation can mimic the behavior of real touching.\n\nIf the pipeline only works with simulation, the applicability will be largely limited.\n\n3.\tOn the writing\nThe writing of this paper is quite clear, especially the description about the difference on the reconstruction network. The whole work is easy to read and follow. \n4.\tOn the significance\nThe integration of vision and touch modalities are important for many applications, especially on robot. 3D shape reconstruction is a key problem when revealing the underlying 3D geometric information from the observations. I think it is a rather under explored area.\n5.\tConclusion\nThough I have some concerns about the technical contribution and the experiments, I also admit I may set a rather high bar. As the vision-touching learning problem is interesting, important and less explored, I am positive on this work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a learning-based approach to 3D reconstruction using a combination of (optional) visual and tactile measurements. A modified neural network takes as input visual and tactile observations and outputs an estimate of the object's 3D shape. This network is coupled with an active perception policy that chooses the next contact point to facilitate tactile-based reconstruction. The method is evaluated through a newly proposed simulator that provides synthetic tactile and visual measurements.

The reviewers agree that the idea of combining tactile observations with visual measurements in an active fashion is a compelling approach to object-scale 3D reconstruction. The paper is well written and easy to follow, and provides an extensive set of experiments that demonstrate the effectiveness of the approach. However, the experiments are limited to simulated data and thus it is not clear how well the framework would perform when operating with real-world visual and tactile measurements. The authors are encouraged to provide a more compelling discussion of the generalizability of the method and, ideally, include real-world experimental results.
Decision: Accept (Poster)