Tactile Image-to-Image Disentanglement of Contact Geometry from Motion-Induced Shear
Authors: Anupam K. Gupta
Laurence Aitchison
Nathan F. Lepora
Venue: CoRL2021 Poster (2021)
Keywords: Robotic Touch
Disentanglement
Shear
Object Reconstruction

Abstract: Robotic touch, particularly when using soft optical tactile sensors, suffers from distortion caused by motion-dependent shear. The manner in which the sensor contacts a stimulus is entangled with the tactile information about the stimulus geometry. In this work, we propose a supervised convolutional deep neural network model that learns to disentangle, in the latent space, the components of sensor deformations caused by contact geometry from those due to sliding-induced shear. The approach is validated by showing a close match between the unsheared images reconstructed from sheared images and their vertical tap (non-sheared) counterparts. In addition, the unsheared tactile images faithfully reconstruct the contact geometry masked in sheared data, and allow robust estimation of the contact pose of use for sliding exploration of various planar shapes. Overall, the contact geometry reconstruction in conjunction with sliding exploration were used for faithful full object reconstruction of various planar shapes. The methods have broad applicability to deep learning models for robots with a shear-sensitive sense of touch.

Tactile Image-to-Image Disentanglement of Contact Geometry from Motion-Induced Shear
Anupam K. Gupta Department of Engineering Mathematics Bristol Robotics Laboratory
Laurence Aitchison Department Computer Science University of Bristol, U.K.
University of Bristol, U.K.
Laurence
ac.
anupam. gupta@rristoo..aoo...o
Nathan F. Lepora
Department Engineering Mathematics
Bristol Robotics Laboratory
University of Bristol, U.K. n. t..epra@@ristto..ccc
Abstract:
Robotic touch, particularly when using soft optical tactile sensors, suffers from distortion caused by motion-dependent shear. The manner in which the sensor contacts stimulus entangled with the tactile information about the stimulus geometry. In this work, we propose supervised convolutional deep neural network model that learns disentangle, in the latent space, the components of sensor deformations caused by contact geometry from those due to sliding-induce induceed shear. The approach is validated by showing a close match between the unsheared images reconstructed from sheared images and their vertical tap (non-sheared) counterparts. In addition, the unsheared tactile images faithfully reconstruct the contact geometry masked in sheared data, and allow robust estimation of the contact pose of use for sliding exploration various planar shapes. Overall, the contact geometry recon- struction in conjunction with sliding exploration were used for faithful full object reconstruction of various planar shapes. The methods have broad applicability deep learning models for robots with a shear-sensitive sense of touch.
Keywords: Robotic Touch, Disentanglement, Shear, Object Reconstruction
2000
Tisk
Clover
Teardrop
Sensor
Tartile Stimuli
DS
(a)
(b)
(d)
Figure Experiment setup and model architecture. (a) TacTip sensor, (b) Sensor internal surface, (c) 3D-printed tactile stimuli with approximate locations of data collection (red circles), (d) Schematic of data collection, showing target (red dot) and initial contact locations (blue dots). Canonical (tapping) data was collected by vertically contacting the target; sheared (sliding) data was collected by sliding to the target, and (e) Schematic the Disentangled UnshearNet model architecture with representative input/output tactile images and contact geometry reconstructions.
Introduction
fundamental aspect of tactile sensing is that measures the environment via physical contact. This entangles the stimulus related tactile information with that induced by the manner of contact between the sensor and the stimulus. For example, when sliding rubbing fingers across tactile stimuli,
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

soft tactile sensors are inevitably distorted by motion-dependent shear, making the sensor response destoryddependent and generalization hard in tactile tasks. This issue is prevalent in camera-based optical tactile sensors that use the lateral motion of markers to infer contact geometry, such as the Tip [1, 2], ChromaTouch [3], GelForce [4] and the optical tactile sensor described in [5], but may occur also in other shear-sensitive tactile sensors. Generally, this shear distortion of the sensor response when sliding along surfaces would make it difficult to perform tactile-dependent tasks such as shape reconstruction and continuous object exploration or manipulation.
Therefore, an important capability is to separate the deformation due to the manner of contact from that due to the stimulus geometry, For example, when continuously exploring an object surface, the sliding-induee induced shear would need to be removed from tactile image to extract the local contact shape. Previous work on extracting pose information from contact continuously slide a tactile sensor over complex objects used deep convolutional neural network trained to map between shear-distorted inputs and the desired (pose) outputs, which was necessary for sliding exploration over a range of complex novel objects [6, 7, 8]. Here, we suggest, an alternate disentanglement based approach that learns to remove motion-induced shear from raw sheared sensor response that has several advantages over previous task-specific approach of training insensitivity to shear, in that: (a) it is computational and data efficient by only requiring the 'unshearing' of the tactile data to be learnt once instead of relearning for each tactile task, such as when applied full object reconstruction (Sec. 4.4); (b) the models can then be learnt more easily unsheared data, which we demonstrate by predicting local object pose; moreover, in some cases no model need be learnt at all, which we demonstrate with contact surface reconstruction using a Voronoi tessellation from the tactile marker data; and (c) the methods are extensible to including other downstream variables of interest appropriate to the task, rather than having to completely retrain new models from scratch.
Our approach to disentangling environmental factors of variation and object attributes in tactile sensing is based on work in computer vision where disentanglement is considered important to improve the generalizability of machine learning models, for example to aid in learning of robust object representations for object classification under novel scenarios [9]. Inspired from this, here we propose a supervised convolutional neural network model that learns to disentangle the components of sensor deformation caused by the contacted object geometry from those due to motion-induced shear. The models were trained using paired data from vertical tap (non-sheared) and continuous (sliding) touch, with an image-to-image Disentangled UnshearNer model architecture that outputs unsheared tactile images from sheared images (Fig. 1). Our validation: (1) quantifies that the unsheared images from continuous motion match their paired tactile images from discrete taps ); (2) demonstrates that local contact geometry reconstructed from unsheared images using Voronoi match their paired tapping counterparts over several stimuli shapes (Fig. 4, top row); (3) infers pose from the sheared tactile images by re-using a convolutional neural network model trained on vertical tap (non-sheared) data, which was then applied to controlling robust sliding exploration around several planar objects (Fig, second row); and (4) combines the two previous steps to attain faithful full-object reconstruction of several planar shapes (Fig. 4, third row).
(Sec. 4. 4)
Background
Measuring strain is a fundamental component of tactile sensing, as emphasised by Platkiewicz, et al [10]: 'the pressure distribution at the surface of a tactile sensor cannot be acquired directly and must be inferred from the [strain] field induced by the touched object in the sensor medium."
There are many mechanisms whereby normal strain and shear strain measured by tactile sensors. Taxel-based tactile sensors such the iCub fingertip [11] and Bio [12] directly measure normal strain. Nevertheless, this measurement of normal strain will be affected by shearing of the soft sensor medium, although the effects on sensor output will be much smaller than those from normal pressure.
Camera-based optical tactile sensors can be very sensitive to shear because they transduce deformation of the soft sensor surface into lateral image. For optical tactile sensors such as the TacTip biomimetic optical sensor used here [1, 2, 14] and other marker- based sensors [3. 5], the surface shear will also distort the apparent geometry of the contact. This distortion is because the normal indentation is

---

represented in the local shear strain of the markers, upon which the global shear strain is superimposed. This contrasts with optical tactile sensors that reconstruct the indentation field from reflection, such as the GelSight [15], where shear will shift the imprint on the tactile image but otherwise the imprint shape should relatively unaffected.
This effect of shear complicates tactile perception under general motions against an object, and restricts the application of touch on challenging tasks such as tactile exploration and surface shape reconstruction. Studies have been limited to discrete contacts where the sensor taps discretely over the object to minimize shear [17, 18, 20), or to rigid planar tactile sensors [21, 22] that do not shear.
recent body of work on the Tac sensor used here focused on mitigating the effects of motion- dependent shear during tactile servo control to slide around planar shapes [6, 19] and complex 3D objects [7, 8]. The shear-dependence problem was addressed in three studies that used deep convolutional neural network to predict pose by training insensitivity to shear into the prediction network [6, 7, 8J. While effective, this has a drawback that procedures for learning shear-invariance needs to be introduced into the training for each individual task, and thus would not generalize easily to new tasks. Another study did not use deep learning, but instead dimensionally reduced the tactile data [19]. where the the pose components naturally separated from the shear components.
This work pursues alternate approach using deep learning to remove the effect of motion-dependee shear from the sheared tactile image, offering several advantages that were summarized in the introduction. Our approach to disentangling environmental factors of variation and object attributes in tactile sensing is based on work in computer vision where disentanglement is considered important to improve the generalization of machine learning models, for example to aid in learning of robust object representations for object classification under novel scenarios [9]. Once disentangled, different environmental factors and object attributes can be recombined to generate coherent novel concepts thus extending knowledge to previously unobserved scenarios [9]. For example, synthesizing sensor responses to novel stimuli by combining the stimuli attributes and factor of variation without actually observing them to plan action under previously unobserved conditions. In the case of tactile sensing considered here, the stimuli attributes could be novel contact geometry and the factors of variation the catiinninnduced shear.
Methods
3.1 Experimental Setup
The setup used in this study was similar to that previously used in [6, 7, 8, 19] where a biomimetic soft optical tactile sensor - the TacTipp was mounted on an industrial robot arm (ABB IRB 120) as an end effector. The sensor skin morphology mimics the layered dermal and epidermal structure of the human fingertip [1. 2, 14] and encodes tactile information in the shear displacement of the papillae (marker) pins, on the inner side of the sensing surface, caused by the skin deformation. The TacTip used in this work consisted of 3D printed soft rubber-like hemispherical dome (40 mm radius, TlngoBlack+) with its inner surface covered with 331 marker pins with white tips arranged in concentric circular grid. The dome was filled with an optically-clear silicone gel to achieve compliance akin to human fingertip. An RGB camera (ELP 1080p module) captures the inner surface of the dome. For more details, we refer to the references above. The manual final assembly of the sensor introduces minor variability the sensor response of the otherwise similar sensors.
For training, we used three 3D-printed (ABS plastic) planar shapes: circular disk. clover and teardrop shown in Fig. (c) (red circles show the approximate location of data collection). For testing, we used sever planar shapes with various morphologies including two acrylic shapes with distinct frictional properties to the other five 3DD-printed shapes (Fig. 4, second row). All shapes were securely fastened to the workspace to prevent accidental motion.
3.2 Data Collection
3.2.1
Paired Canonical and Sheared Data
This work proposes supervised deep learning model Disentangled UnshearNer remove the effect of motion-induced global shear in the tactile images, caused by the friction between the sensor skin
3

---

and its contacted surface during contact. We emphasise this motion- indiond shear is different from the skin deformation due to local shear from the geometry of the stimulus imprint.
Our method requires paired tactile data with minimal global shear taken from vertically tapping onto the stimulus (referred to canonical data) and sliding data with random global shear (referred as sheared data). The sheared data is collected after sliding randomly and is paired with tactile image from the canonical data by using the same relative poses between the sensor and the stimuli (Fig. 1 (d)). The raw tactile data was cropped, sub- sub-sampled to 256 256-pixel region and finally converted to binary images using an adaptive threshold to minimise sensitivity to internal lighting conditions.
In total, data for 200 distinct poses were collected for each of the three training stimuli (Fig. 1 (c)). To generate poses, the sensor location relative to the target contact location (red circle, Fig. 1 (d)) was sampled randomly from uniform distribution spanning the range [-5, 5] mm in the two lateral directions (along the x- and y-axes), [-45, -45] deg in yaw (0nz) and [-6, -1] mm in depth (z-axis). Ranges of the target poses and perturbations to generate canonical and sheared data were chosen to ensure safe contact of the sensor with the stimulus (Fig. 1 (c)).
The canonical data was taken by vertically tapping onto the stimulus, to give a representative sample of tactile data with minimal global shear just due to the stimulus geometry. This gives target (reference) data that the sheared data from sliding contacts should be restored to. The dataset, in total, had 30,000 samples: 50 instances of cach of the 200 poses recorded previously for each of the three stimuli shapes (Fig. (c)). To generate instances, the indentation depth (z-axes) was varied randomly between [-1, 1] mm from the original indentation depth.
Canonical Data:
The sheared data accords canonical tactile image distorted by global shear from lateral sliding motion of the sensor on the stimulus. Thus, the sheared data does not give contact-only information akin to canonical data that represented the geometry of the contacted stimulus, but has significant component due to the contact motion.
Sheared Data:
The dataset, in total, had 90,000 samples: 150 instances of each of the 200 random poses recorded previously for each of the three stimulus shapes (Fig. 1 (c)). To generate instances for each pose, the lateral position offset for sliding the target pose was varied randomly between [-2.5, 2.5] mm in lateral direction(s) (along the x- y-axes or both), using the procedure shown in Fig. 1 (d).
Training and Test Data: The canonical and sheared datasets were partitioned randomly into training, validation and test sets (ratio 60:20:20); i.c. from the 200 unique poses for each stimulus shape (Fig. I (c)), 120 poses were assigned to a training set and 40 poses each to validation and testing sets. Overall the canonical and sheared training set contained 18,000 and 54,000 samples respectively, while the validation and testing sets contained 6,000 and 18,000 samples respectively. To generate paired canonical and transformed data for training, each transformed tactile image was paired randomly with one of the canonical instances having the same pose.
3.2.2 Pose-labelled Data
To train pose prediction network PoseNet, separate dataset was collected instead of reusing the dataset used to train Entangled & Disentagled UnshearNet. This was done to prevent the model from overfitting the training set and ensure performance of the trained UnshearNer models despite potential dataset shifts due to a host of factors, including drift in sensor characteristics over time or with continued use due to wear and tear of the tactile sensing surface.
The dataset had, in total, 30,000 samples: 10,000 samples from each of the three training object shapes (Fig. (c)). To generate contact poses, the sensor location relative to the initial contact location was sampled randomly from a uniform distribution spanning the range [-5, 5] mm laterally (x- and y-axis directions), [-45, -45] deg in yaw (Onz) and [-6, -1] mm in depth (z-axis). Ranges of the target poses, as before, were chosen to ensure safe contact of the sensor without causing damage. As discussed above, all tactile images were cropped, sub-sampled and thresholded to 256 x 256-pixel to binary (black/white) tactile images. The dataset was partitioned randomly into training, validation and test sets in the ratio 60:20:20 as above, containing 18,000, 6,000 and 6,000 samples respectively.

---

Encoder (E)
Decoder (D)
I.2 Loss
PC
32 64
64
128 128
128
64
64
32
PC
LI Patch Loss
Convolution (K: 4x4, S:2)
Transposed Convolution (K: 4x4, S:2)
tactile image was processed by the encoder E and decoder D with output PC: the unsheared counterpart of input post removal of sliding-induce distortion. The match between predicted unsheared data PC and canonical data C was enforced by training with direct comparison between PC and along with comparing corresponding patches (20 x 20 pix.) taken from PC and C to enforce local compliance.
Figure 2: Entangled UnshearNet architecture: A
Convolation (K: 4x4, S:2)
Transposed Convolution
Docoder (DC)
(K: 4x4, 8:2)
L2 Loss
Pose Code
128 64
PC
64
L.1
Patch Loss
Shear Code
Shear Decoder (DS)
64
PS
Reconstruction
128
64
Figure 3: Disentangled UnshearNet architecture: Labelling as in Fig. 2, with two decoders DC and DS. The first decoder has output PC: the unsheared counterpart of input post removal of sliding-induced distortion; the second decoder DS reconstructed the input S. denoted PS.
3.3 Image-to-Image Models for Shear Removal
To achieve our goal of removing the effects of shear induced by the sliding motion, we trained two models based on the encoder-decoder framework Entangled UnshearNet (Fig. 2) and Disentangled UnshearNet (Fig. 3). The former model (baseline) learns the mapping between the input sheared data and its corresponding canonical (tap) data directly. The later model instead, first disentangles the contact-only (pose) sensor response component from the sliding-induced global shear component in the latent space of the encoder E; followed by accurate reconstruction of the input sheared image and its corresponding canonical (tap) image using decoders DS (sheared or sliding) and DC (canonical or tapping). We hypothesized that more robust mapping between sheared and canonical (tap) data can be learned by disentangling the contact-only and sliding-indde induced components of sensor response. The mathematical formulation of the Disentangled UnshearNet is listed below:
The encoder E splits the latent space into two components z == (Zp. E Zp x Z, given paired canonical { and shear-transformed training samples {x, i-11 where x E X, and x, E X, are in the sets of canonical and sheared data, respectively.
The decoder DC takes input only the pose part of the latent code Zp and reconstructs the canonical sample corresponding the input sheared sample. In combination with the encoder E, it thus leams mapping M X, + X from the sheared to canonical data.
5

---

The other decoder DS, takes as input the full latent code and reconstructs the input sheared sample. In combination with the encoder E, it thus acts as an auto-encoder and learns a mapping N X - X from the sheared data to itself.
Our objective function has three components:
(1) L2 loss for matching the generated canonical samples M(xx) to their targets xe;
(2) LJ patch loss for matching random patches extracted from generated canonical images M and corresponding targets X, for enforcing image similarity locally as changes in sensor response due to change contact conditions are predominately local nature; (3) Reconstruction loss for matching the generated sheared samples (xx)) to their target input sheared samples Xy.
The objective functions are:
Lice (E, = Ee, ||| - (S,
(1)
Lup (E, = D
DD((2))
(2)
Leachh(E (C,DC) = Xx, (xxer)ps(xx) (ropp()(()))))
(3)
where Zs) = (x²) for all X, E X, The function crops() selects a patch of size 20 20 pix from M(x,) and Xe. The objective functions Lee- Lxup and Lpatch represent sheared reconstruction loss,
canonical supervised loss and canonical supervised patch loss respectively.
The final objectives for Encoder E and Decoders DC, DS are:
Le = Tu (E,DS) DS) + sup (E, DC) + 2 paptch (E, DC),
(4)
T_pc sup DC) + pucch (E: DC),
(5)
Lps Lrc DS),
where 2 0.1 is the relative loss scale factor.
Details of the specific model architectures and their training are given in Appendices A & B.
4
Experimental Results
4.1 Shear Removal from Tactile Images
We first performed an ablation study to demonstrate the separation latent representations by the Disentangled UnshearNet into pose and shear codes respectively. For details see Appendix D.
The first test of the effectiveness of our approach in removal of motion-induced global shear used multi-scale structural similarity index (SSIM) [26] to compare image similarity between Disentangled (Entangled) model-generated unsheared images PC and their test set sheared and canonical (non- sheared) C counterparts. On comparison, we found the Disentangled / Entangled UnshearNet model generated unsheared images PC to be structurally closer to the target canonical images C (93% 90%) than their sheared S counterparts (32%). This clearly shows that our model Disentagled UnshearNet performed better in removing global shear from tactile images vis-á-vis the baseline model Entangled UnshearNet.
4.2 Reconstruction of Local Contact Geometry
Motion-induced global shear masks the contact geometry necessitating removal of shear from tactile images to reconstruct true indentation field (Fig. 4, top row). We used Voronoi-basee method introduced in Cramphorn al [13) reconstruct indentation field from tactile data encoded by TacTip It transforms the displacements of the tactile markers the inner side of sensing surface into areas of hexagonal cells tessellating the grid of the markers [13, Fig. 4]. The change in Voronoi cell areas from an undeformed reference represent the magnitude of local skin distortion caused by contact, which correlates with the indentation due to contact geometry.
For display, we fit 3D surface to the (x.y) centroid coordinates using the Voronoi cell areas as the corresponding height values (Fig. 4, top row). The fitted surface test set sheared data S is visibly distorted by shear (Fig. 4, top row). However, once the global shear is removed successfully by

---

Twp
Sbearel
Uniibarte
Ealangled)
(Dieatangled)
Unsitared
Top
Sheared Undhrated
(Ratnogled)
Undieeree
(Hieslungled)
Top
Sleeared
Undieared
(Fatzogled:
Unshearad
(harnlunglet)
Spina
Disk
Star
Clover
Teardnop
Figure 4: Contact shape reconstructions, shape exploration and reconstruction. Top row: Local contact geometry reconstruction. The approximate contact geometry can be recovered from sheared data using the from the original sheared data. Middle row: Trajectories, overlaid on objects, under robust sliding using pose estimation from Disentangled UnshearNel generated unsheared images. Red dot is the starting point of trajectory. Bottom row: Full object reconstructions from combining the unsheared local contact geometry reconstructions along the sliding trajectories. High-resolution figures are in Appendix E.
unsheared images generated by Disentangled UnshearNer but not
Disentagnled UnshearNei in model-gen genee-eeee unsheared images PC, the fitted surface closely matches with the true representation of the indentation field obtained from the canonical (non-sheared) tap) data (Fig. 4, top row). Also clear from Fig. 4 (top row) that model Disentangled UnshearNet is more effective in removing global shear vis-á-vis the baseline model Entangled UnshearNet.
4.3 2D Sliding Shape Exploration
We further confirmed the performance of our model on removing global shear by computing the mean-absolute error (MAE) between the predicted pose (lateral position tx and in-plane orientation Onz) from Disentangled Entangled UnshearNet generated unsheared images PC, sheared images S and canonical (non-sheared / tap) images C and the target pose using PoseNet trained only on canonical
vertical tap) images (Table 1). On comparison, the lowest prediction error was found on unsheared samples obtained using disentangled network. The error increased by as much as 50% and 5x higher on unsheared samples from entangled network and original sheared samples respectively. Again, the UnshearNet models managed to successfully correct the distortion in sensor response cause by slide-induced global shear, with the disentangled representation giving the best performance.
Table 1: Mean Absolute Error (MAE) in Pose Predictions
Tap Data
Sheared Data
Unsheared Data (Entangled)
Unsheared Data (Disentangled)
horizontal, &, (mm)
0.43
2.72
0.95
0.64
yaw, Onz (degrees)
2.13
22.20
6.73
4.38
To test the real-time performance and model generalizability to novel stimuli, used PoseNer for sliding exploration of several planar shapes (Fig. 4, second row). This testing included four additional planar shapes absent from training including two acrylic shapes that differed in frictional properties from the other 3D printed stimuli (spirals & 2, Fig. 4, second row). The robot successfully traced contours across all stimuli shapes (Fig. 4, second row) using a combination of Disentangled UnshearNet for global shear removal, PoseNer for prediction of object pose from gener--geeerate unsheared images and a simple servo control policy (see Appendix C). This clearly indicates that the proposed Disentangled UnshearNet is well suited for correcting the sliding-induce thear-disorrte sensor response for real-time servo control applications.

---

4.4 Object Shape Reconstruction
As a final test of our model's utility to remove shear, we reconstructed full object shapes by combined use of contact geometry estimation (Sec. 4.3) and sliding shape exploration (Sec. 4.4), both tasks adversely affected by motion-induced shear (Fig. third row). To attain full object reconstructions, we fused together the contact information extracted from unsheared images corresponding sliding contacts (recorded during shape exploration) and linearly interpolated them over rectangular grid (Fig. 4, third row). These results clearly demonstrate the effectiveness of our approach in learning the sheared-unshearer mapping that can then later be reused for multiple downstream tasks. In addition, the faithful shape reconstructions along with the shape exploration across multiple novel planar objects show that the model successfully generalizes to novel situations.
5 Discussion
This work proposed supervised approach to remove the distortion in sensor response caused by motion-induced global shear while preserving sensor deformations induced by the spatial geometry of the contacted stimuli. The proposed approach was based on an encoder-decoder structure and learned to remove shear from tactile images by disentangling the pose (contact only) and shear in the latent space (Fig. 3). The disentanglement of sensor components contributed due to contact only and global shear promoted learning of robust sheared-unsheud mapping as demonstrated by better performance of this approach vis-á-vis the baseline (Fig. 2).
This approach was validated by: (1) demonstrating good match (93%) between model-generated unsheared images (from sliding contacts) and its canonical (non-sheared / vertical tap) counterparts using structural similarity index measure (Sec 4.1). (2) faithful reconstruction of contact geometry, masked in sheared images, from model-generated unsheared images (Fig. 4, top row). (3) predicting the local object pose from model generatee unsheared images with prediction network trained only on canonical (non-sheared) / vertical tap) data (Table 1). This was later used for successful sliding exploration of several planar objects (Fig. 4, second row). 4) combining contact geometry reconstruction and sliding exploration to reconstruct full object shapes for several planar objects (Fig, 4, third row) which remains an problem under active investigation in the field [24, 25).
Our approach improves upon related work [6, 7] that instead trained insensitivity to shear into the prediction network itself. While effective, this approach does not extend readily to predict other tactile dimensions such as parameters of the local contact geometry. In that case, new labelled data would need collecting to train new model for each intended task. Our approach, of shear removal from tactile images, instead means that single model based on the canonical (tapping) data can be re-used.
The current study has several limitations: 1) considered only planar objects. We, however, expect to extend readily to 3D objects similar to recent results that have successfully extended PoseNet to complex 3D surfaces (3 pose components) and 3D edges (5 pose components) for sliding explo- ration [6, 7J. 2) tested only on TacTip optical tactile sensor. The approach should hold well for other soft optical sensors although difference in frictional and elastic properties between different sensor types will necessitate retraining of models. 3) training was limited to translation shear even though servo control introduces rotational shear due to sensor rotation while sliding over objects. The fact that the methods worked well both for sliding exploration and contact geometry reconstruction even in absence of explicit compensation for rotational shear, shows that some unshearing carries over from translation rotational shear least for planar objects. Our expectation is that bringing rotational shear into the training will become important when extending the methods to 3D objects. and 4) requires paired canonical and sheared data, which complicates the data collection. An approach that is more economical with labelled data requirements would be a worthwhile extension of the current work.
Overall, we expect the methods developed here can can be applied more widely to various other exploration and manipulation tasks involving soft tactile sensors that are adversely affected by motion-induced shear.
8

---

Acknowledgements
Anupam Gupta and Nathan Lepora were supported by a Research Leadership Award from the Leverhulme Trust on 'A biomimetic forebrain for robot touch' (RL-2016-39).
References
[T] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and
N. F. Lepora," The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies," Soft Robot. 2018 Apr:5(2:211-2277
[2] N. Lepora," " Soft Biomimetic Optical Tactile Sensing With the TacTiip Review," in IEEE Sensors Journal, vol. 21, no. 19, pp. 21131-21 143, Oct. 2021.
[3] X. Lin, L. Willemet, A. Bailleul and M. Wiertlewski, "Curvature sensing with a spherical tactile sensor using the color-interference of a marker array," 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 603-609.
[4] K. Kamiyama, K. Vlack, T. Mizota, H. Kajimoto, N. Kawakami, and S. Tachi. Vision-based sensor for real-time measuring of surface traction fields. in IEEE Computer Graphics and Applications, vol. 25, no. 1, pp. 68-75, Jan.-Feh. 2005.
[5] C. Sferrazza and D' Andrea, Design, Motivation and Evaluation of a Full-Resolution Optical Tactile Sensor," Sensors, vol. 19(4), pp. 928, February 2019.
[6] N. F. Lepora, A. Church, C. de Kerckhove, R. Hadsell and J. Lloyd, "From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and Optical Biomimetic Tactile Sensor," in IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 2101-2107, April 2019.
[7] N. Lepora and J. Lloyd, "Optimal Deep Learning for Robot Touch: Training Accurate Pose Models of 3D Surfaces and Edges," in IEEE Robotics & Automation Magazine, vol. 27, pp. 66-77, June 2020.
[8] N. F. Lepora and Lloyd. Pose-based tactile servoing: Controlled soft touch using deep learning. IEEE Robotics & Automation Magazine, pp. 2-15, 2021.
[9] Y.Bengio. Deep Learning of Representations: Looking Forward. In: Dediu AH., Martín C., Mitkov R., Truthe B. (eds) Statistical Language and Speech Processing. SLSP 2013. Lecture Notes in Computer Science, vol 7978. Springer, Berlin, Heidelberg, 2013.
[10] Platkiewicz, Lipson, and Hayward. Haptic Edge Detection Through Shear. Scientific Reports, 6(1):23551, September 2016.
[11] N. Wettels, V. Santos, R. S. Johansson, and G. E. Loeb. Biomimetic Tactile Sensor Array. Advanced Robotics, 22(8):829 889, January 2008.
[12] A. Schmitz, Maiolino, M. Maggiali, L. Natale, G. Cannata, and G. Metta. Methods and Technologies for the Implementation Large-Scale Robot Tactile Sensors. IEEE Transactions on Robotics, 27(3):389 June 2011.
[13] I.. Cramphorn, J. Lloyd and N. F. Lepora, "Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations," 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018 pp. 2752-2757.
[14] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, "Development of Tactile Sensor Based on Biologically Inspired Edge Encoding," in 2009 International Conference on Advanced Robotics, 2009, pp. 1-6.

---

[15] W. Yuan, S. Dong, and E. H. Adelson, "GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force," Sensors, 17(12):2762, 2017.
[16] H. Liu, K.C. Nguyen, V. Perdereau et. al., "Finger contact sensing and the application in dexterous hand manipulation," Auton Robot, 39, 25-41, (2015).
[17] N. Lepora, K. Aquilina and I.. Cramphorn, "Exploratory Tactile Servoing With Active Touch," " IEEE Roboties and Automation Letters, vol. no. 2, pp. 1156-1163, April 2017.
[18] U. M.-Hernandez, T. J. Dodd, M. H. Evans, T. Prescott, and N. F. Lepora, "Active sensorimotor control for tactile exploration. Robotics and Autonomous Systems, 87, pp. 15-27, 2017.
[19] K. Aquilina, D. A. W. Barton and N. F. Lepora, "Shear-invariant Sliding Contact Perception with Soft Tactile Sensor," 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 4283-4289.
[20] C. Lu, J. Wang and S. Luo, "Surface Following using Deep Reinforcement Learning and GelSight Tactile Sensor", arXiv:1912.007455/ R[c..RO), December 2019.
[21] Q. Li, C. Schurmann, R. Haschke, and H. Ritter, "A control framework for tactile servoing," " Roboties: Science and Systems, 2013.
[22] Z. Kappassov, JA.C. Ramon and V. Perdereau, "ZMP Features for Touch Driven Robot Control via Tactile Servo," in International Symposium on Experimental Robotics, Springer, 2016, pp.
234-243.
[23] D.P. Kingma and J.L. Ba." Adam: A method for stochastic optimization," 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, 2015.
[24] S. Suresh, M. Bauza, K. Yu, J, G. Mangelson, A. Rodriguez and M. Kaess, "Tactile SLAM: Real-time inference of shape and pose from planar pushing." " 2021 IEEE International Confer- ence on Robotics and Automation (ICRA), Xi an, China, 2021, pp. 11322-11328.
[25] M. Bauza, O. Canal and A. Rodriguez (2019). Tactile mapping and localization from high- resolution tactile imprints. In 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 381 1831177.
[26] Z. Wang. P. Simoncelli and A. C. Bovik, "Multiscale structural similarity for image quality assessment," The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, 2003, pp. 1398- 1402 Vol.2.
10


Reviews:
[{"review_id": "dBwheP-Dj12", "reviewer": "Reviewer_JxBB", "summary": "The paper proposes a supervised learning method to remove artifacts from optical-tactile signals for robot manipulation tasks. Specifically, the artifacts in question are caused by sensor distortion in the presence of motion-induced shear. In the approach, the model is trained on tactile images collected from un-sheared contact with a few distinct canonical shapes. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is generally well written and easy to follow\n- Experiments are detailed and the approach is clearly defined.\n- Results seem promising.\n\nWeaknesses\n- The scope of the paper strikes me as fairly narrow, both in experiment and comparison to existing methods.\n- Why does the approach not consider task-specific training, such as predicting local surface geometry directly for surface mapping (as opposed to an intermediate un-sheared representation) ? It would be nice to highlight the benefits of this approach when compared to approaches which model local contact geometry, such as the following (I have no affiliation):\n- Bauza, Maria, et al. \"Tactile object pose estimation from the first touch with geometric contact rendering.\" arXiv preprint arXiv:2012.05205 (2020).\n- There seems to be a lack of baselines in the approach. How might this compare to using an analytical model, for instance? \n- The scope of the experiments seem quite narrow. Only three shapes were investigated, and all were made from the same material. To demonstrate that the method is indeed learning a meaningful representation (as opposed to over-fitting to data), it would be useful to use test objects and motions which were not observed in training. \n- There does not seem to be any discussion of rotational vs translational shear. I would think that the former would be more challenging to disentangle than the latter, but this should be discussed and investigated.\n-  A description and picture of the TacTip sensor would be very helpful to the reader.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "A significant contribution to the community would require broadening the scope of the paper, clearly demonstrating the generality of the approach comparison to existing state-of-the-art methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Please address the points listed above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "aooPyJ8WSNp", "reviewer": "Reviewer_xr1d", "summary": "This work proposed to disentangle the contact geometry from shear, which can benefit sliding motion and manipulation for soft tactile sensors. The authors proposed the encoder-decoder network trained from paired supervised data. The system is evaluated to reconstruct unsheared canonical images, local shape reconstruction. It is further applied for contour following with sliding motion.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The disengagement of the contact geometry and shear is meaningful, and can benefit future applications with shear-related tactile and manipulation tasks.\n- The author proposed a network and automatic data collection framework to tackle the problem, using paired supervised data.\n- The disentangled results are evaluated for contour following.\n\nWeaknesses:\n- The comparison between Entangled and Disentangled UnshearNet seems unfair. The Disentangled UnshearNet clearly has more supervision, in the autoencoder fashion to reconstruct the raw image. A fair comparison could be also adding raw input reconstruction to the Entangled network. By current comparison, it is unclear whether the improvement is from the additional supervision or the disentangled network flow.\n- Additionally, there seems to be no clear evidence that the embedding of the Disentangled network is disentangled into “Pose Code” and “Shear Code”. What if the “Pose Code” contains both “deformation” and “shear” information, and “Shear Code” contains minor/part of shear information? It seems that the network flow doesn’t have the supervision to force the embedding to be disentangled, since the “Pose Code” is used for both geometry and raw input reconstruction. And there is no validation of what the “Shear Code” learns. One experiment would be using the “Pose Code” and “Shear Code” to predict the motion that induced the shear. It can also be added as the supervision to the network to force “Shear Code” alone to learn the motion. \n- For 3.2.2 Pose-labelled data, the data collection procedure seems very similar to the one for canonical data. Is there a reason for collecting a separate dataset?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper demonstrated an important problem of entangled shear for contact geometry. The framework with the supervised paired data solved the problem well. But the technical part of the network structure requires more experiments to support its claim of disentanglement of “Pose Code” and “Shear Code” for the Disentangled UnshearNet.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Quote from the weaknesses:\n- The comparison between Entangled and Disentangled UnshearNet seems unfair. The Disentangled UnshearNet clearly has more supervision, in the autoencoder fashion to reconstruct the raw image. A fair comparison could be also adding raw input reconstruction to the Entangled network. By current comparison, it is unclear whether the improvement is from the additional supervision or the disentangled network flow.\n- Additionally, there seems to be no clear evidence that the embedding of the Disentangled network is disentangled into “Pose Code” and “Shear Code”. What if the “Pose Code” contains both “deformation” and “shear” information, and “Shear Code” contains minor/part of shear information? It seems that the network flow doesn’t have the supervision to force the embedding to be disentangled, since the “Pose Code” is used for both geometry and raw input reconstruction. And there is no validation of what the “Shear Code” learns. One experiment would be using the “Pose Code” and “Shear Code” to predict the motion that induced the shear. It can also be added as the supervision to the network to force “Shear Code” alone to learn the motion. \n- For 3.2.2 Pose-labelled data, the data collection procedure seems very similar to the one for canonical data. Is there a reason for collecting a separate dataset?\n"}}, {"review_id": "GkfXcCFJmP", "reviewer": "Reviewer_gcAq", "summary": "The paper develops a learning paradigm to obtain non sheared tactile images from shear contact.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is attempting to solve an interesting problem which benefits the domain of visuotactile sensors for tactile perception. The illustrations in figures bring clarity to the approach. There are some concerns with the paper in its current form which are listed below:\n\n1. One major concern with the approach is that it’s not clear to me whether the approach works on unseen objects. Also, it was not clear whether the learned model is object dependent. Conducting experiments on unseen objects would help bring clarity.\n\n2. Does the model generalize across sensors or do we need to train per sensor?\n\n3. Additionally, the paper mentions an appendix but it was not submitted. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would like to know whether the model generalizes before I can make a better recommendation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Addressing the below issues will improve the paper:\n\n1. It’s not clear whether the model was trained separately for each object or if a single model was trained across objects.\n\n2. Running experiments on unseen objects.\n"}}, {"review_id": "AMP1HKzP4o3", "reviewer": "Reviewer_oHk8", "summary": "The paper presents a method for handling the motion-dependent shear effect in tactile sensors. In particular, as objects can move under soft tactile sensors, tactile signal is distorted or sheared, which leads to difficulties in employing the resulting tactile signal, for example when performing object pose estimation. The paper proposes to learn “unshearing” deep neural network models that can map from sheared tactile images to canonical non-sheared images. Two architectures are proposed: entangled architecture that directly learns to reconstruct the canonical image from the sheared one, and a disentangled architecture that decomposes the intermediate representation into canonical and shear-specific parts that are trained to reconstruct the canonical and the original sheared tactile images respectively. Finally, the unsheared images are used to estimate object poses using a pose-model trained on the canonical non-sheared images. The paper collects a paired dataset of sheared and canonical tactile data to train the unshearing models. The resulting models are evaluated on three objects and TacTip biomimetic tactile sensor. They are shown to produce unsheared tactile images with a high similarity to the canonical ones and successfully reconstruct the original geometry of the objects, as well as significantly improve object pose prediction, with the disentangled model having a higher performance than the entangled one. Pose prediction is also used to perform servo control of robot following contours of the objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Reducing the shearing effect of tactile sensors is important for real-world application of tactile signals.\n- Disentangling reconstruction into sheared and non-sheared parts is an elegant way to improve unshearing performance.\n- The paper is well-written and easy to understand and follow.\n- The method is shown to work on a real TacTip biomimetic tactile sensor and produce a high level of reconstruction accuracy.\n\nWeaknesses:\n- Although the dataset is split into training and test parts, all the data still comes from the same three objects. It would be useful to see performance of the method on some novel objects to better understand the generalization capability of the method.\n- Right now, the PoseNet is trained on canonical images and evaluated on sheared/unsheared images. It would be interesting to see results of training the PoseNet directly on the sheared images if such a pairing is possible. This would help to better understand the advantage of reconstructing the unsheared image for pose prediction. \n- It would be interesting to see more details on the robotic control system used to perform the contour following experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper shows that it is possible to unshear tactile sensor signals, which is important for many real-world robotic applications. The paper could benefit from some additional experiments to demonstrate the generalization capabilities of the method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to \"Strengths And Weaknesses\" for a detailed list of issues.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a machine learning model to disentangle the effects of motion-induced shear from contact geometry in vision-based tactile sensors.

Quality:

(-) All reviewers have concern that the experimental validation does not fully support the claims of the manuscript. I here highlight the concerns from Reviewer gcAq and JxBB regarding the generalization capabilities of the proposed approach to unseen objects and sensors, and the additional experiments suggested by Reviewer xr1d to validate the claims about the Disentangled UnshearNet. Performing the experiments requested by the four reviewers would benefit the overall quality of the paper. 

(+) Real-world experiments

Clarity:

(+) The paper is generally clear and well written

(/) The appendix is not included

Originality:

(+) The proposed approach is novel 

Significance:

(+) Several reviews agree that the topic is relevant and impactful

Additional comment: From a philosophical perspective, it should be better argued why Disentangling Contact Geometry from Motion-Induced Shear is useful. If you consider that this is a component of a wider system (e.g., control), a devil's advocate might ask why not using an end-to-end learning approach (together with the downstream task) in the first place and side-step the need to explicitly disentangle.


---

After rebuttal:
The authors included additional experiments and answered the reviewers in such a way as to address most of the concerns.
While the paper is likely to impact a very niche part of the robotic community, the contribution appears to be solid.
Decision: Accept (Poster)