# Visual-Tactile Fusion for 3D Objects Reconstruction from a Single Depth View and a Single Gripper Touch for Robotics Tasks

Mohamed Tahoun, Omar Tahri, Juan Antonio Corrales Ramon, Youcef Mezouar

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mohamed Tahoun, Omar Tahri, Juan Antonio Corrales Ramon, Youcef Mezouar. Visual-Tactile Fusion for 3D Objects Reconstruction from a Single Depth View and a Single Gripper Touch for Robotics Tasks. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021), Sep 2021, Prague, Czech Republic. pp.6786-6793, 10.1109/IROS51168.2021.9636150. hal-03558917

HAL Id: hal-03558917

https://hal.science/hal-03558917v1

Submitted on 14 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Visual-Tactile Fusion for 3D Objects Reconstruction from a Single Depth View and a Single Gripper Touch for Robotics Tasks

Mohamed Tahoun<sup>1,2</sup>, Omar Tahri<sup>3</sup>, Juan Antonio Corrales Ramón<sup>4</sup>, and Youcef Mezouar<sup>2</sup>

<figure><img src="image_3.png" /><figcaption>Fig. 1: Pipeline of reconstruction operation.</figcaption></figure>

**Abstract**— The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object's shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object's occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects' reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object's shape using a single arbitrary depth-view and a single dexterous-hand's touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 64<sup>3</sup> for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (40<sup>3</sup>, 64<sup>3</sup> and 128<sup>3</sup>) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.

## I. INTRODUCTION

A lot of research is being conducted nowadays in developing robotic platforms that can perform successful autonomous manipulation tasks, e.g. assembly and disassembly

tasks in factories, housework assistance, moving objects, etc [1]. It is believed that 3D object reconstruction could provide a safe, fruitful and autonomous interaction between robots and the robots' space through recognizing and understanding objects to be manipulated [2]. Generally, 3D object reconstruction research depends on visual sensing [2][3]. Unfortunately, visual sensors have limitations when dealing with automated real life applications due to different lighting conditions and object-self occlusions [4]. These limitations present challenges when reconstructing a full 3D object. Hence, using another sensory modality such as tactile sensing could definitely help to overcome these challenges as it is extensively used in robotic platforms. Especially, if both sensors were to be used together like for humans.

Frequently, depth map from an arbitrary single viewpoint does not give sufficient data to precisely anticipate the object's full 3D shape due to the uncertainty of occluded regions based on prior work [5]. Therefore, the tactile information was employed to solve this uncertainty by touching the object from the occluded side to have a more accurate estimation of the complete object. Visual-tactile fusion is a very challenging task as it deals with two different types of sensor as well as the obtained information from depth sensors is very large relative to the obtained one from tactile sensors. In the present work, depth and tactile sensors are employed as a multi-modal sensory system for 3D shape perception problems. The proposed method is based on visual-tactile fusion to predict a more accurate full 3D shape of the object. More precisely, a 3D-CDCWGAN (3 Dimensional Conditional Deep Convolutional Wasserstein Generative Adversarial Network) was designed and trained to perform 3D shape reconstruction that could help robots anticipate future

<footnote n="1">¹ INSA Centre Val de Loire, PRISME Laboratory EA4229, F-18000 Bourges, France mohamed.tahoun@insa-cvl.fr</footnote>

<footnote n="2">² Université Clermont Auvergne, CNRS, Clermont Auvergne INP, Institut Pascal, F-63000 Clermont-Ferrand, France youcef.mezouar@sigma-clermont.fr</footnote>

<footnote n="3">³ VIBOT EMR CNRS 6000, ImViA, Université Bourgogne Franche-Comté (UBFC) omar.tahri@u-bourgogne.fr</footnote>

<footnote n="4">⁴ Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, 15782, Santiago de Compostela, SPAIN juanantonio.corrales@usc.es</footnote>
actions like path planning and generating stable grasps. The proposed model was trained on the two inputs with the goal to get good balance between both sensors' information. This is achieved through having a separate branch for each modality with a fusion at the feature level.

To summarize, the contributions of this work are: 1) A novel CNN architecture for 3D shape reconstruction from a single arbitrary depth view and a single hand's touch. 2) An end-to-end trainable multi-modal semi-supervised generative deep learning model of multiple household objects. 3) Balanced visual-tactile fusion for more accurate object's 3D shape reconstruction. 4) A large dataset of aligned depth, tactile and ground truth voxel grids with different dimensions (40³, 64³ and 128³'), that is based on the kinematic model of the Shadow dexterous hand.

This paper is structured as follows. Section II addresses the related works. Section III describes the proposed GAN reconstruction method and the proposed visual-tactile dataset development as well as the network architecture. Followed by the training methodology and the used evaluation metric in section IV. Then, section V presents the experimental results with a discussion on the reconstruction results. Finally, conclusions and future works are described in section VI.

## II. RELATED WORKS

Grasping unknown objects is commonly limited to using the point cloud from a single view. Therefore, getting the complete 3D shape would be worthwhile to plan complex manipulation tasks. 3D shape completion from depth maps or partial scans has been studied widely in robotics, computer vision, and computer graphics [2][3][5]. Multi-modal sensors have been commonly used in robotics for tracking [6], grasping [7], scene exploration with a Gaussian Process [8] and objects' shape recognition using active exploration [9][10]. Specifically, tactile data has been used for 3D object completion and reconstruction tasks [11].

Tactile information has been used to supplement visual ones for 3D shape reconstruction. For example, Bjorkman et al. [12] reconstruct the object models from visual and tactile measurements using Gaussian process implicit surfaces for manipulation tasks. The objects were reconstructed based solely on visual information, then the object is touched 12 times to update the reconstruction. Gaussian processes have a high computational cost proportional to the number of measurement points cubed which affects the manipulation task's execution time. Ilonen et al. [13] fused visual and tactile information while grasping the object to reconstruct the complete 3D object using an iterative extended Kalman filter given that the object is symmetric. Differently from the previous works, this paper proposes a 3D object shape reconstruction using a Deep Learning scheme with a single dexterous-hand's touch (low computational complexity) and aimed for all kinds of object (symmetric and asymmetric). In the proposed model, a single hand's touch alongside a depth image were utilized to influence the entire reconstructed object. It is distinct from Wang et al. [14]'s model where

several touches were utilized in their model to update the visual completion in proximity to the touched points.

## III. THE PROPOSED APPROACH

### A. Overview

Fig. 1 presents an overview of the presented 3D reconstruction system. This approach is proposed to reconstruct a full 3D object surface using a partial representation of the object from two modal sensors based on a deep learning technique. This partial representation represents the fusion of features' vectors of depth and tactile information. The system starts by collecting a single partial depth view of the object using an RGB-D camera. Then, the contact information is obtained by moving the robotic hand to the object's occluded side to perform a single touching action on the object using a multi-fingered dexterous hand. Straightaway, depth sensor's data are immediately segmented and voxelized in voxel grid $V_D$, as well as the calculated contact points obtained using the hand's tactile sensors are voxelized in voxel grid $V_T$. Then, both $V_D$ and $V_T$ multi-modal inputs' voxel grids are used as the reconstruction model's input. Finally, the system's output is the estimated voxel grid $V_P$ of the full 3D shape. In a voxel grid, the occupied voxel contains 1 while the empty one contains 0, which is also known as an occupancy map or volumetric grid. The output voxel grid $V_P$ is normally aligned with the input voxel grids $V_D$ and $V_T$. This proposed reconstruction approach is based on a system which includes an RGB-D sensor (Intel® RealSense™ depth camera D435 [15]) and a five-fingered robotic hand (The Shadow Dexterous Hand [16]) that is equipped with SynTouch BioTac tactile sensors [17] on each finger.

### B. Visual-Tactile Dataset

Having a broad visual-tactile dataset for learning-based reconstruction tasks is challenging since the majority of the open-source datasets are generated solely from depth maps. Watkins-Valls et al. created a dataset [18] based on ground truth meshes from the Grasp [19] and YCB [20] datasets. It consists of half a million point cloud's triplets of depth information $X_{cf}^{depth}$, sparse randomly-generated tactile points $X_{cf}^{tactile}$, and ground truth $GT_{ob}$ plus the transformation matrix from the camera frame to the object frame $CTM_{cf}^{ob}$. They synthesized 726 depth images for each object through placing the object's ground truth in Gazebo [21] and applying uniformly sampled rotations around the mesh in roll-pitch-yaw, (11×6×11). They randomly generated the tactile points for each depth image by sampling voxels along the occluded side of the ground truth's voxel grid. It is impractical to employ these randomly generated points with the Shadow dexterous hand, since they are not bounded by the hand's kinematic properties. To tackle this issue, the proposed aligned depth-tactile dataset has its generated tactile information constrained to the hand's kinematics, which it has been produced based on Watkins' dataset.

In order to generate the tactile data for each sample, first, the depth $X_{cf}^{depth}$ and ground truth $GT_{ob}$ information should be aligned on the same reference frame. For that, the
<figure><img src="image_4.png" /><figcaption>Fig. 2: Simulation on generating tactile sample for the proposed dataset.</figcaption></figure>

transformation matrix $CTM_{cf}^{ob}$, which was supplied in the dataset for each sample, was used to transform the mesh ground truth into the depth camera frame. Then both of them were voxelized to obtain $V_{GT}$ and $V_D$. An unoccupied tactile voxel grid $V_T$, which will contain the tactile data, was created with the same dimensions as $V_{GT}$. As a result, the ground truth's occupied voxel grid $V_{GT}$ was obtained automatically aligned with the partial depth view grid $V_D$ and the $V_T$ as well. Furthermore, the ground truth's occluded side (opposite view to the camera) is considered as a squared $2-D$ plane. To retrieve the tactile points, it was assumed that the object will be touched by the hand near to its center. For that, a region of interest (ROI) with a high certainty of fingers touching the object was selected and the rest was neglected. On another word, the ROI in the ground truth's $2-D$ plane was selected from the occluded side of the object by selecting 80% of the area of the predefined $2-D$ plane. Then, a random $(x,y)$ point was sampled to generate the tactile data provided that they are within the boundaries of the ROI. Where this $(x,y)$ point refers to a potential contact spot for the first finger of the hand when it touches the object as shown in Fig. 2. The locations of the rest of hand's fingers' are estimated constrained on the kinematic diagram of the Shadow hand in [16]. Next, a ray was traced in the $-Z$ direction (towards the depth sensor in Fig 2) through the approximate positions of fingers in $V_{GT}$ and the first occupied voxel was stored as a tactile information in $V_T$. For each tactile sample of training data, this process was repeated 6 times to accumulate 6 random touches of the hand to the object which correspond to 24 contact points between the fingers and the object. In contrast, for each tactile sample of test data, only 4 contact points are obtained to match a real test case with a single hand's touch to the object. Finally, it was verified that the depth and tactile voxel

grids are well overlapped with the ground truth voxel grids with threshold of 80%. In other words, if the overlapping percentage between the depth $V_D$ and tactile $V_T$ voxel grids and the ground truth $V_{GT}$ voxel grid is less than 80%, this sample is neglected from the dataset.

The present dataset consists of 608 objects, each sample in each object consists of an aligned depth map, tactile points and ground truth occupancy grids where $V_D$, $V_T$ and $V_{GT}$ are oriented in the camera frame. These occupancy grids were generated by Binvox [22] and were stored with different occupancy grids resolutions $40^3$, $64^3$ and $128^3$. The input voxel grid set is $V_D$ and $V_T$. Whereas, $V_D$ is the depth occupancy grid that is filled only with the regions of the object visible to the camera, $V_T$ is the tactile occupancy grid that is filled with some possible touches of the fingers of the hand in the occluded side. While the output voxel grid set $V_{GT}$ is the ground truth occupancy grid that is occupied by the whole 3D object. In particular, the occupancy grids of $64^3$ were used in the proposed approach as detailed in section III-C.

### C. Network Architecture

In this approach, an advanced 3D deep generative convolutional model 3D-CDCWGAN has been implemented. The goal is to get a high-resolution good reconstruction with the capability of generalization and to overcome the high resolution's memory limitation. Both 3D-CDCWGAN model's inputs ($V_D$ and $V_T$) and output ($V_P$) are voxel grids of a resolution $64^3$. In particular, the model role is trying to learn a mapping function $f$ that maps the depth and tactile inputs to a full 3D object as in (1).

$$V_P = f(V_D, V_T) \quad (1)$$

where $V_D$ and $V_T \in Z^{64^3}$, and $Z$ can be $\{0, 1\}$. This function $f$ is generally characterized through the convolutional layers [23]. The present network generally follows the idea of the 3D-RecGAN [24] that is based on Conditional Generative Adversarial Networks (cGANs) [25]. In general, the basic Generative Adversarial Networks (GAN) [26] consists of a generator and a discriminator. Where the discriminator tries to discriminate between the real outputs and the synthesized outputs from the generator. In contrast, the generator aims to generate outputs from a random latent space that are indistinguishable from the real ones to maximally confuse the discriminator. As a matter of fact, cGAN is a simple extension of the basic GAN which allows the model to condition on external information (i.e. object ground truth) rather than generating from a random latent space. This makes the mapping function $f$ in cGAN less complex than the original GAN.

The generator in the 3D-CDCWGAN model is based on the autoencoder with U-net architecture [27] that is followed by a conditional discriminator. Instead of reconstructing the initial input and discovering an optimal encoding, the network's generator (autoencoder) attempts to learn the association between full 3D structures and partial 3D structures of depth and tactile data. The autoencoder will learn a
function *f* defined in (1) and produce a plausible 3D object given new depth and tactile data through the monitoring of full 3D objects. However, the reconstructed 3D objects are grainy and lack fine detail [24] during testing the autoencoder without training the discriminator. This issue was overcome during the training process though feeding the autoencoder's reconstructed 3D object into the conditional discriminator to ensure its plausibility. In detail, $V_D$ and $V_T$ are paired once with the corresponding full 3D object ground truth $V_{GT}$ and defined as “real sample pair” denoted by $(V_D \oplus V_T) \sqcup V_{GT}$ where $\oplus$ represents a concatenation operation. Then, inputs are paired once again with the generated 3D object $V_P$ and defined as “fake sample pair” denoted by $(V_D \oplus V_T) \sqcup V_P$. Subsequently, discriminating fake sample pairs against true sample pairs is the discriminator's goal.

While GANs can provide visually pleasing samples, they are complex to train due to the difficulty in converging the Jensen-Shannon divergence-based loss function [26]. A lot of recent work [28] has been done on this issue for finding ways to optimize learning, however stable and reliable GANs' learning remains an open dispute. The recent WGAN [29] leverages Wasserstein distance with weight clipping as a loss function which helps to stabilize the training procedure. Furthermore, the extended research WGAN-GP [30] improved the training process of the original GAN. In detail, WGAN-GP loss function uses a gradient penalty with respect to its input for guaranteeing fast and stable convergence. Fig. 3 presents the detailed architecture of the proposed multi-modal reconstruction model where the generator and the discriminator are located in the figure's upper and lower blocks.

1) **The generator:** is based on an autoencoder that has two encoders with skip-connections between encoders and decoder. There is one encoder for depth data and the other for tactile data. Unlike a simple GAN generator such as the

vanilla GAN that generates data from noisy latent distributions, the proposed multi-modal reconstructions generator synthesizes data from the latent distribution of the fused depth and tactile information. In detail, each encoder consists of five 3D convolutional layers. Then a batch normalization (BN) layer that's followed by a leaky ReLU activation function plus a 3D max Pooling layer. The number of the first convolutional layer filters starts with 64, then doubles in each consecutive layer and ends with 512. Then, each encoder is pursued by a two fully connected (FC) and BN layer. At that point, the depth and tactile features vectors are fused together by concatenating the depth and tactile FC layers output before applying the second layer's BN and ReLU activation function. FC layers and the fusion process objective is to enclose the latent space with the semantic depth and tactile information that will be followed by the decoder. The decoder is composed of 5 mirrored de-convolutional layers. Then a BN layer that is accompanied by a ReLU function for the first 4 layers and a BN with a sigmoid function for the last layer. The up-sampled feature maps are concatenated channel-wise with feature-maps of similar size from the encoding path before each deconvolution operation. These concatenation connections allow the use of high resolution features previously learned in the network during the up-sampling process. Typical autoencoders without the two fully connected layers and the skip U-Net connections are unable to learn plausible full 3D shapes due to the limited latent space and the local structure's absence. Furthermore, using a separate encoder for each sensor modality and the fusion emplacement guarantee having balanced weights for both modalities' information. Notably, the generator is supervisely trained by providing the 3D shape ground truth.

2) **The discriminator:** takes as input both real and fake sample pairs. It aims to discern the plausibility of the reconstructed 3D objects from the multi-modal autoencoder. It has

<figure><img src="image_5.png" /><figcaption>Fig. 3: 3D-CDCWGAN Model architecture</figcaption></figure>
five 3D convolutional layers identical to the convolutional layers of the generator encoder. Pursued by a BN layer accompanied by ReLU activation function for the first 4 layers and a BN layer with a sigmoid activation function for the last convolutional layer. Discriminator in simple GAN and cGAN is designed as a binary classifier of fake samples among real samples. When back-propagating in the generator, two classes will not be reasonable to learn the object's geometric details due to the high dimensional distributions (3 × 64³) of the discriminator input (fake sample or real sample). Thereupon, the 3D-CDCWGAN's discriminator task is to distinguish between the distributions of latent representations of fake and real reconstructions that is represented by the discriminator's output 1D latent vector. Whereas, the 3D-CDCWGAN's generator is trained to make these two distributions as identical as possible.

### D. Loss Functions and Optimization

Both, 3D-CWGAN's generator and discriminator has a loss function to be optimized. Generator's loss $\mathcal{L}^G$ is obtained by,

$$ \mathcal{L}^G = \omega \mathcal{L}_{AE}^G + (1 - \omega) \mathcal{L}_{GAN}^G \quad (2) $$

$$ \mathcal{L}_{GAN}^{G}(\theta^{G}, \theta^{D}) = -\mathbb{E}[D(\hat{y}|x)] \quad (3) $$

where $\mathcal{L}_{GAN}^G$ (3) is the loss of generating fake reconstructed pairs, $\mathcal{L}_{AE}^G$ (4) is the object reconstruction loss for the autoencoder, and $\omega$ is a weight optimizer between both of them.

$$ \begin{aligned} \mathcal{L}_{AE}^{G} &= E(y, \hat{y}) \\ &= -(\beta y \log(\hat{y}) + (1 - \beta)(1 - y) \log(1 - \hat{y})) \end{aligned} \quad (4) $$

where $\beta$ is the BCE balancing factor and $y \in \mathbb{R}^m$ is the flattened version of the ground truth voxel grid $\mathbb{V}_{GT} \in \mathbb{R}^{n^3}$. Similarly, $\hat{y} \in \mathbb{R}^m$ is the flattened version of the predicted voxel grid $\mathbb{V}_P \in \mathbb{R}^{n^3}$. Hence, it is noted that $m = 262144$ elements of the flattened 1D vector and $n = 64$ voxels in each dimension. While, each value $y_i$ of y can be $\{0, 1\}$, and the values of $\hat{y}_i$ of $\hat{y}$ are real numbers bound in (0.0001, 0.9999) instead of (0, 1). In fact, this adjustment diminishes the probability of vanishing gradients because the gradient's loss magnitude increases over the $\hat{y}$ range. The standard BCE equally weights the false positive and false negative outcomes. Also, the number of unoccupied voxels is higher than occupied voxels' number. So, the network is prone to making false positive estimates and can easily dive into a local optima. Therefore, the standard BCE was modified by adding a balancing factor $\beta$ [31]. It adjusts the balance between the false positives and false negatives instead of making them equally important through imposing a higher penalty on false positive results than on false negatives ones. In contrast, the discriminator loss function in $\mathcal{L}_{GAN}^D$ (5) is calculating the difference of fake sample distributions against real ones.

$$ \mathcal{L}_{GAN}^{D}(\theta^{G}, \theta^{D}) = \mathbb{E}[D(\hat{y}|x)] - \mathbb{E}[D(y|x)] + \lambda \mathbb{E}[(\|\nabla_{\hat{y}}D(\hat{y}|x)\|_2 - 1)^2] \quad (5) $$

where $\tilde{y} = \epsilon y + (1 - \epsilon)\hat{y}$, $\epsilon \sim U[0, 1]$, $\lambda$ is the gradient penalty coefficient of WGAN-GP loss [29][30].

To conclude, the whole 3D shapes are best learned when $\mathcal{L}_{AE}^G$ is minimized as it boosts the output $\hat{y}$ to tend to be either 1 for occupied target voxels and 0 for empty target voxels. Whereas reconstructing more reasonable 3D shapes conditioned on the fused depth and tactile input is achieved when $\mathcal{L}_{GAN}^G$ is reduced. Also, the discriminator performance in distinguishing fake samples and real samples is improved when $\mathcal{L}_{GAN}^D$ is minimized.

## IV. EXPERIMENTAL SETUP

### A. Training and Tuning

An end-to-end training process is maintained for the whole network from scratch which is implemented using Python with PyTorch1.4.0 and TensorFlow1.14 libraries. Both the generator and discriminator were optimized simultaneously. The chosen optimizer was Adam [32] for both of them with the default hyper-parameters ($\beta_1=0.9, \beta_2=0.999, \epsilon=10^{-8}$) with batch size of 32. And “He normal”[33] was used to initialize the model kernel weights. During training, $\beta$ was set to 0.9 for the modified BCE, $\omega$ was set to 0.001 and $\lambda$ was set to 10 as recommended in the original paper. It was found that best learning rates for this model should be from $7 \times 10^{-5}$ to $7 \times 10^{-4}$ for the generator, while from $2 \times 10^{-5}$ to $2 \times 10^{-4}$ for the discriminator. Accordingly, the generator and the discriminator learning rates were set to $5 \times 10^{-4}$ and $1 \times 10^{-4}$ respectively. Notably, these parameters were selected after several trials of different learning rates, batch sizes, and epochs. Moreover, 8 NVIDIA Tesla V100 GPUs were used for training the network.

### B. Dataset Employment

The developed visual-tactile dataset consists of 608 objects, where 75% of the objects were used for both training and validation while the rest for testing. In this way, test objects are certainly never seen by the model during the training process. To train the proposed model, it is provided the dataset triplets of aligned arbitrary single depth views $x_{depth}$, tactile points obtained from hand's touch to the object $x_{tactile}$, and a ground truth $y$ of a 3D object. The proposed 3D-CDCWGAN model performed best for real test scenarios, when 6 hand touch's action of the object were taken into account for the training samples. The validation and testing samples are considered as a real test scenario because their tactile's samples are generated by a single hand touch's action of the object.

### C. Evaluation metrics

Intersection over Union (IoU), Volume Similarity (VS) and Average Hausdorff Distance (AVD) metrics were used to evaluate the performance of the 3D-CDCWGAN reconstruction model between the predicted voxel grid $\mathbb{V}_P$ and the ground truth voxel grid $\mathbb{V}_{GT}$. Each metric's detailed description, motivation and derivatives can be found in [34].
1) Intersection over Union (IoU): is also known as Jaccard Similarity or Jaccard score.

$$ 
\begin{aligned}
IoU &= \frac{|\mathbb{V}_P \cap \mathbb{V}_{GT}|}{|\mathbb{V}_P \cup \mathbb{V}_{GT}|} = \frac{|\mathbb{V}_P \cap \mathbb{V}_{GT}|}{|\mathbb{V}_P| + |\mathbb{V}_{GT}| - |\mathbb{V}_P \cup \mathbb{V}_{GT}|} \\
&= \frac{\sum_{i=1}^{m} (\hat{y}_i * y_i)}{\sum_{i=1}^{m} (\hat{y}_i + y_i) - \sum_{i=1}^{m} (\hat{y}_i * y_i)}
\end{aligned} 
\quad (6) $$

where $\hat{y}_i$, $y_i$ and $m$ are the same as defined in III-D. The higher the IoU value, the better the used reconstruction model.

2) Volume Similarity (VS): is a measure that considers the volumes of the predicted and ground truth voxel grids to indicate similarity. VS is defined as $1 - VD$ as follows [34].

$$ VS = 1 - \frac{FN - FP}{2TP + FP + FN} \quad (7) $$

where FN, FP and TP are false negatives, false positives and true positives between $\mathbb{V}_P$ and $\mathbb{V}_{GT}$. The higher the VS value, the better the used reconstruction model.

3) Average Hausdorff Distance (AVD): is a widely used metric to calculate the distance between two point sets X and Y.

$$ AVD(X, Y) = \left( \frac{1}{X} \sum_{a \in X} \min_{b \in Y} d(a, b) + \frac{1}{Y} \sum_{b \in Y} \min_{x \in X} d(b, a) \right) / 2 $$

It can be written in a more simplified way as follows:

$$ AVD(\mathbb{V}_P, \mathbb{V}_{GT}) = \left( \frac{GTtoP}{GT} + \frac{PtoGT}{P} \right) / 2 \quad (8) $$

where $GTtoP$ is the directed average Hausdorff distance from ground truth to prediction, $PtoGT$ is the directed average Hausdorff distance from prediction to ground truth, $GT$ is the number of voxels in the ground truth, and $P$ is the number of voxels in the prediction. The lower the AVD value, the better the used reconstruction model.

# V. EXPERIMENTAL RESULTS AND DISCUSSION

## A. Batch Normalization and U-Net experiments

To achieve the desired performance in reconstruction tasks, the presented model was empirically selected by testing different configurations as specified in Table I. During the configuration phase, the number of epochs and the batch size were fixed to 15 and 32 as well as the number of training samples was fixed to 4096 (i.e 128 batches).

<table><caption>TABLE I: Different configurations for 3D-CDCWGAN</caption><thead><tr><th></th><th>Config.1</th><th>Config.2</th><th>Config.3</th><th>Config.4</th></tr></thead><tbody><tr><th>BN</th><td>False</td><td>False</td><td>True</td><td>True</td></tr><tr><th>U-Net</th><td>False</td><td>True</td><td>False</td><td>True</td></tr></tbody></table>

In detail, each configuration was tested with two groups of learning rates for the generator and the discriminator from the mentioned range in section IV-A (LR Group 1 and LR Group 2). LR Group 1 was set to ($lr_G^1 = 9 \times 10^{-5}$ and $lr_D^1 = 4 \times 10^{-5}$) while LR Group 2 was set to ($lr_G^2 = 4 \times 10^{-4}$ and $lr_D^2 = 9 \times 10^{-5}$). Then, each configuration has been tested separately to get the IoU's average on the 4096 training

<table><caption>TABLE II: IoU of training and testing views on different configurations with LR group 1 and group 2 (Larger is better)</caption><thead><tr><th rowspan="2"></th><th rowspan="2">U-Net</th><th rowspan="2">BN</th><th colspan="2">LR Group 1</th><th colspan="2">LR Group 2</th></tr><tr><th>Train</th><th>Test</th><th>Train</th><th>Test</th></tr></thead><tbody><tr><th>Config.1</th><td>False</td><td>False</td><td>0.660</td><td>0.586</td><td>0.654</td><td>0.496</td></tr><tr><th>Config.2</th><td>False</td><td>True</td><td>0.681</td><td>0.550</td><td>0.673</td><td>0.554</td></tr><tr><th>Config.3</th><td>True</td><td>False</td><td>0.756</td><td>0.635</td><td>0.793</td><td>0.629</td></tr><tr><th>Config.4</th><td>True</td><td>True</td><td><strong>0.796</strong></td><td><strong>0.660</strong></td><td><strong>0.803</strong></td><td><strong>0.678</strong></td></tr></tbody></table>

samples, and on all the testing views for each configuration.

From Table II, it can be seen that “Config.4” is the best configuration which includes BN layers and skip U-Net connections in its architecture. In fact, BN between CNN layers was used to normalize the inputs of each layer, where the input from prior layers could be changed after the weights update. Also, the use of BN accelerates the training process because it helps the model to train in fewer epochs. Equally important, the skip U-Net connections between the encoder and the decoder guaranteed the diffusion of the fused input voxel grid’s local structure.

## B. Comparison to baseline methods

In this section, the multi-modal 3D-CDCWGAN visual-tactile reconstruction model is compared to the visual-tactile proposed approach by Watkins-Valls et al. [18] and against 3D-CDCWGAN that is trained using a single modality information. Firstly, Watkins developed a deep learning model to reconstruct the 3D object from a single 2.5D depth view and more than 4 tactile points which correspond to several touching actions to the object [18]. Their model is based on a typical 3D-CNN architecture which has three 3D-CNN layers and two fully connected layers that encourage the reconstruction performance [35]. Differently, the presented 3D-CDCWGAN is trained to reconstruct 3D objects from a single 2.5 depth view and only 4 tactile points which correspond to a single dexterous-hand’s touch. Plus the fact that it is based on a generative model that is believed capable of providing better results. Their network’s output is limited to a 40³ voxel grid that is not extensible to higher resolutions due to the high computing cost of the fully connected layers, while the present model is a 64³ and it can be extended to higher resolutions. For this experiment, Watkins’s model was trained using the developed dataset with the model’s original configurations in [18]. To have a fair comparison with them, the input voxel grids are downsampled to 40³ and the generated voxel grids are upsampled to 64³ voxel grids using trilinear interpolation to match the dataset information and the present model output. Secondly, one 3D-CDCWGAN was trained on tactile data, and it performed poorly owing to the lack of tactile points. Also, another 3D-CDCWGAN was trained on depth data only which is the same as 3D-RecGAN [24] from the state of the art. It outperforms the tactile only model since the depth maps have a lot of points. However, it experienced many conditions where there was inadequate data to correctly estimate the occluded part of the object. On the contrary, the employment of both sensory
information in the 3D-CDCWGAN depth and tactile model were successfully used to estimate plausible geometries of the occluded areas.

<table><caption>TABLE III: IoU, VS and AVD of validation and testing views for depth-tactile Watkins-Valls et al. [18]'s model plus the depth only and depth-tactile 3D-CDCWGAN model</caption><thead><tr><th rowspan="3">Views</th><th colspan="2">Wakins et al. [18]</th><th colspan="4">3D-CDCWGAN</th></tr><tr><th colspan="2">Depth & Tactile</th><th colspan="2">Depth only</th><th colspan="2">Depth & Tactile</th></tr><tr><th>Valid.</th><th>Test</th><th>Valid.</th><th>Test</th><th>Valid.</th><th>Test</th></tr></thead><tbody><tr><td>IoU</td><td>0.685</td><td>0.607</td><td><strong>0.840</strong></td><td><strong>0.799</strong></td><td><strong>0.845</strong></td><td><strong>0.811</strong></td></tr><tr><td>VS</td><td><strong>0.950</strong></td><td>0.925</td><td>0.942</td><td><strong>0.929</strong></td><td><strong>0.948</strong></td><td><strong>0.932</strong></td></tr><tr><td>AVD</td><td>2.369</td><td>2.502</td><td><strong>0.953</strong></td><td><strong>1.031</strong></td><td><strong>0.811</strong></td><td><strong>0.908</strong></td></tr></tbody></table>

Table III validates the performance of the proposed model. It also confirms the disparity between the 3D-CDCWGAN depth only model versus the depth-tactile model by Watkins in terms of IoU, VS and AVD averages for each model on validating and testing views. Where the 3D-CDCWGAN depth only model is performing better than the depth-tactile model by Wakins which proves the superiority of the 3D-CDCWGAN model's architecture. Although Watkins' model gives better reconstruction in terms of VS for validating views, 3D-CDCWGAN depth and depth-tactile models outperformed their model for the testing objects that have never been seen during training.

### C. Qualitative results

Fig. 4 shows qualitative reconstruction results on depth and tactile samples of testing objects using Watkins' model and both the 3D-CDCWGAN depth and depth-tactile models. It is found that the depth only and depth-tactile 3D-CDCWGAN reconstructions are better than Watkins' ones. Not only using GANs in such a reconstruction task could improve the reconstructed output rather than using a 3D-CNN deep learning model. Besides, the presence of the tactile points improves the reconstruction from the occluded side. In fact, the object boundaries are closely defined as well as the tactile data is well employed in distinguishing among the various potential reconstructions. This multi-modal model proves that the tactile information could reconstruct more accurate 3D shapes when the depth data alone is inadequate to reconstruct accurate ones.

## VI. CONCLUSION

A novel 3D Convolutional Wasserstein GAN multi-modal model has been proposed to reconstruct the complete 3D shape of an observed object from an arbitrary single view and a single hand's touch. The model has been trained and tested on a developed dataset where the tactile points have been generated based on the hand's kinematic model. The obtained results have shown that the proposed scheme has the capability of the generalization when testing on new objects. The employed approach gives a promising reconstruction performance that outperforms state-of-the-art methods. Future work will be devoted to test the obtained reconstructed models for grasping tasks by using them for

<figure><img src="image_6.png" /><figcaption>Fig. 4: Qualitative results of reconstructions from different approaches on new objects. a) Inputs: Depth (Red) & Tactile (Blue) Inputs. b) Results from Watkins-Valls et al. [18]. c) Results from depth only model. d) Result from the proposed 3D-CWCGAN depth-tactile model. e) Original ground truth.</figcaption></figure>

grasp planning and estimating the grasping points. There is also an interest in developing an exploration algorithm for the occluded region using the hand.

## ACKNOWLEDGMENT

This work has received funding from the French government research program "Investissements d'Avenir" through the IDEX-ISITE initiative 16-IDEX-0001 (CAP20-25), and the European Union's Horizon 2020 research and innovation program under grant agreement n° 869855 (Project SoftManBot). Additionally, the authors are grateful to the Mésocentre Clermont Auvergne University for providing computing resources.

## REFERENCES

[1] C.-A. Smarr, T. L. Mitzner, J. M. Beer, A. Prakash, T. L. Chen, C. C. Kemp, and W. A. Rogers, "Domestic robots for older adults: Attitudes, preferences, and potential," *International journal of social robotics*, vol. 6, no. 2, pp. 229-247, 2014.

[2] J.Varley, C.DeChant, A.Richardson, J.Ruales, and P.Allen, "Shape completion enabled robotic grasping," in *IROS*, 2017, pp. 2442-2447.

[3] A. Sharma, O. Grau, and M. Fritz, "Vconv-dae: Deep volumetric shape learning without object labels," *ArXiv*, vol. abs/1604.03755, 2016.

[4] X.-F. Han, H. Laga, and M. Bennamoun, "Image-based 3d object reconstruction: State-of-the-art and trends in the deep learning era," *IEEE TPAMI*, 2019.

[5] M. Tahoun, C. M. Mateo, J. Corrales-Ramón, O. Tahri, Y. Mezouar, and P. Gil, "Visual completion of 3d object shapes from a single view for robotic tasks," 2019, pp. 1777-1782.
[6] G. Izatt, G. Mirano, E. Adelson, and R. Tedrake, "Tracking objects with point clouds from vision and touch," in *IEEE ICRA*, 2017, pp. 4000–4007.

[7] P. Allen, A. Miller, P. Oh, and B. Leibowitz, "Integration of vision, force and tactile sensing for grasping," 1999.

[8] J. Bohg, M. Johnson-Roberson, M. Björkman, and D. Kragic, "Strategies for multi-modal scene exploration," in *IEEE/RSJ IROS*, 2010, pp. 4509–4515.

[9] P. Allen, "Sensing and describing 3-d structure," in *Proceedings. 1986 IEEE ICRA*, vol. 3, 1986, pp. 126–131.

[10] P. Falco, S. Lu, A. Cirillo, C. Natale, S. Pirozzi, and D. Lee, "Cross-modal visuo-tactile object recognition using robotic active exploration," in *2017 IEEE ICRA*, 2017, pp. 5273–5280.

[11] Z. Pezzimenti, C. Reyda, and G. D. Hager, "Object mapping, recognition, and localization from tactile geometry," in *IEEE ICRA*, 2011, pp. 5942–5948.

[12] M. Björkman, Y. Bekiroglu, V. Högman, and D. Kragic, "Enhancing visual perception of shape through tactile glances," in *IEEE/RSJ IROS*, 2013, pp. 3180–3186.

[13] J. Ilonen, J. Bohg, and V. Kyrki, "Fusing visual and tactile sensing for 3-d object reconstruction while grasping," in *2013 IEEE ICRA*, 2013, pp. 3547–3554.

[14] S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson, "3d shape perception from monocular vision, touch, and shape priors," in *IEEE/RSJ IROS*, 2018, pp. 1606–1613.

[15] "Intel d435," https://www.intelrealsense.com/depth-camera-d435/.

[16] "Shadow dextrous hand technical specification," https://www.shadowrobot.com/wp-content/uploads/shadow_dexterous_hand_technical_specification_E1_20130101.pdf.

[17] "Biotac-brochure.pdf," https://www.syntouchinc.com/wp-content/uploads/2017/01/BioTac-Brochure.pdf.

[18] D. Watkins-Valls, J. Varley, and P. Allen, "Multi-modal geometric learning for grasping and manipulation," in *2019 IEEE ICRA*, 2019, pp. 7339–7345.

[19] D. Kappler, J. Bohg, and S. Schaal, "Leveraging big data for grasp planning," *2015 IEEE ICRA*, pp. 4304–4311, 2015.

[20] B. Çalli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. Dollar, "The ycb object and model set: Towards common benchmarks for manipulation research," *ICAR*, pp. 510–517, 2015.

[21] N. Koenig and A. Howard, "Design and use paradigms for gazebo, an open-source multi-robot simulator," in *IEEE/RSJ IROS*, 2004, pp. 2149–2154.

[22] P. Min, "binvox," http://www.patrickmin.com/binvox, 2004 - 2019, accessed: 2020-06-10.

[23] M. Valueva, N. Nagornov, P. Lyakhov, G. Valuev, and N. Chervyakov, "Application of the residue number system to reduce hardware costs of the convolutional neural network implementation," *Mathematics and Computers in Simulation*, vol. 177, pp. 232–243, 2020.

[24] B. Yang, H. Wen, S. Wang, R. Clark, A. Markham, and A. Trigoni, "3d object reconstruction from a single depth view with adversarial learning," *2017 IEEE ICCVW*, pp. 679–688, 2017.

[25] M. Mirza and S. Osindero, "Conditional generative adversarial nets," *ArXiv*, vol. abs/1411.1784, 2014.

[26] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, "Generative adversarial nets," in *NIPS*, 2014.

[27] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in *MICCAI*, 2015.

[28] B. Poole, A. A. Alemi, J. Sohl-Dickstein, and A. Angelova, "Improved generator objectives for gans," *arXiv preprint arXiv:1612.02780*, 2016.

[29] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein gan," *ArXiv*, vol. abs/1701.07875, 2017.

[30] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, "Improved training of wasserstein gans," *arXiv preprint arXiv:1704.00028*, 2017.

[31] A. Brock, T. Lim, J. M. Ritchie, and N. Weston, "Generative and discriminative voxel modeling with convolutional neural networks," *arXiv preprint arXiv:1608.04236*, 2016.

[32] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *CoRR*, vol. abs/1412.6980, 2014.

[33] K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," in *IEEE ICCV*, Dec 2015, pp. 1026–1034.

[34] A. A. Taha and A. Hanbury, "Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool," *BMC Medical Imaging*, vol. 15, no. 1, p. 29, Aug 2015.

[35] E. Shelhamer, J. Long, and T. Darrell, "Fully convolutional networks for semantic segmentation," *IEEE TPAMI*, vol. 39, pp. 640–651, 2017.