MidasTouch: Monte-Carlo inference over distributions across sliding touch
Authors: Sudharshan Suresh
Zilin Si
Stuart Anderson
Michael Kaess
Mustafa Mukadam
Venue: CoRL 2022 Oral (2022)
Keywords: Tactile perception
Localization
3D deep learning

Abstract: We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface. This framework takes in posed tactile images over time, and outputs an evolving distribution of sensor pose on the object's surface, without the need for visual priors. Our key insight is to estimate local surface geometry with tactile sensing, learn a compact representation for it, and disambiguate these signals over a long time horizon. The backbone of MidasTouch is a Monte-Carlo particle filter, with a measurement model based on a tactile code network learned from tactile simulation. This network, inspired by LIDAR place recognition, compactly summarizes local surface geometries. These generated codes are efficiently compared against a precomputed tactile codebook per-object, to update the pose distribution. We further release the YCB-Slide dataset of real-world and simulated forceful sliding interactions between a vision-based tactile sensor and standard YCB objects. While single-touch localization can be inherently ambiguous, we can quickly localize our sensor by traversing salient surface geometries. Project page: https://suddhu.github.io/midastouch-tactile/

MoucsTouch: Monte-Carlo inference over distributions across sliding touch
Sudharshan Suresh 1.2 Zilin Si', Stuart Anderson', Michael Kaess', Mustafa Mukadam 'Carnegie Mellon University, Meta Al
https://suddhu, gi thub. www.mi/astotcch-tattiitooooooaoe
Abstract: We present MidasTouch, tactile perception system for online global localization of a dision-based touch sensor sliding on an object surface. This framework takes in posed tactile images over time, and outputs an evolving dis- tribution of sensor pose on the object's surface, without the need for visual priors. Our key insight to estimate local surface geometry with tactile sensing, learn compact representation for and disambiguate these signals over long time horizon. The backbone of MidasTouch is Monte-Carlo particle filter, with a mea- surement model based tactile code network learned from tactile simulation. This network, inspired by LIDAR place recognition, compactly summarizes local surface geometries. These generated codes are efficiently compared against a pre- computed tactile codebook per-object, to update the pose distribution. We further release the YCB-Slide dataset of real-world and simulated forceful sliding inter- actions between a vision-boe -bsoed tactile sensor and standard YCB objects, While single-touch localization can be inherently ambiguous, we can quickly localize sensor by traversing salient surface geometries.
Keywords: Tactile perception, Localization, 3D deep learning
Local 3D grometry
Tactile network
Tictile cods
Tactile se cam
Heightmap
Masa model
X
5012) lookup
codebook
Hinger-object interaction
Motion model
Particle tilter
Sesser
Prior bolief N...
Finger-objert global
Figure MidasTouch performs online global localization of vision-based touch during sliding interactions. Given posed tactile images over time, this system leverages local surface geometry within a nonparametric particle filter to generate an evolving distribution of sensor pose on the object's 's surface.
object surface
1 Introduction
Interactive perception the Catch-22 [1] of robotics; vision can be used to track objects for down- stream contact-rich interactions, but such interactions can occlude visual tracking. In particular, end-effector-objeet relative positioning is crucial for contact-rich policies like sliding [2-6]. in-hand manoeuvres [7-10], finger-gaiting [1]], and multi-fing mulli-finger enclosure [12]. Additionally, vision of- ten affected by object transparency, specularity, and poor scene illumination. With high-dimensional vision-based tactile sensors [13-15], we now have a window into local object interactions. While tactile images from these sensors capture local surface geometry, they lack global context necessary for relative pose tracking. We address this challenge with MidasTouch an online tactile perception system that tracks the evolving pose distribution of vision-basee touch sensor sliding across known objects. We acquire global context by integrating local observations over long horizons.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

The form-factor of Aision-based tactile sensors has restricted prior methods to small parts [16-18] or local tracking [19, 20]. For everyday objects, we can leverage compositionality [21]; each is made up of local geometries that are spatially fixed relative to one another. Consider a simple mug: made up of a curved body, flat base, rounded handle, and sharp lip. Without global context, a single-touch ambiguous: a perceived sharp edge could lie anywhere along the lip of the mug- Such likelihood distribution is spread across the object's 's surface and not unimodal, but interaction over long time horizons disambiguate it. This mirrors haptic apprehension, the exploratory procedures humans perform when presented with familiar object [22].
We approach this as an analogue to mobile robot Monte-Carlo filtering [23], but instead apply on the surface manifold of the object. Just as a mobile robot has access to detailed floor plans, odometry, and cameras, manipulators have access to object meshes, end-effector poses, and vision-based touch. This applicable environments with known object models like households, warchouses, and factories, further facilitated by large-scale scanned object datasets [24, 25]. While priors from vision can bootstrap our system [26]. they not a prerequisite for global estimation.
In addition to open-sourcing MidasTouch, we release a comprehensive real-world and simulated dataset of sliding DIGIT [15] interactions across standard YCB objects [24] with ground-truth. This serves as both an evaluation of MidasTouch, and as a benchmark currently lacking in the tactile sensing community. Both are accessed from our project website. Specifically, our contributions are:
1. Online particle filtering over posed tactile images for a distribution of finger-object poses,
Learned embeddings for vision- based touch using local surface geometry,
The YCB-Slide dataset of DIGIT and YCB object interactions for evaluation and benchmarking.
Our framework relies on recent developments in tactile sensing, rendering, and robotics: (i) vision- based tactile sensors [13-15, 27-31] like the GelSight and DIGIT have the requisite spatial acuity to discern local geometric features, (ii) tactile simulation [32-34] with realistic rendering enables learning tactile observation models and precomputing sppeccf- interactions with sizeable data, which would be infeasible in the real-world, and (iii) learned models for 3D place recognition [35- 37] spawned by ubiquitous LIDAR and RGB-D data that can be extended to tactile sensing.
2 Related work
Binary contact sensors have been used in conjunction with particle filters for global estimation of robot hands relative to simple geometries [38-4] (38-433. These methods require a large amount of touches, but serve
Tactile pose inference:
touchstone for Monte-Caria methods that have seen great success in mobile robotics [23, 44]. With tactile arrays, local patch measurements are integrated for planar estimation [45, 46] or for 3D alignment [47]. The synergy of vision and touch gives much needed global context [48, 49] but is outside our current work's 's scope.
With vision-based touch, Li al. [16] show planar small-part localization by directly computing the homography on GelSight heightmaps. Relative pose-tracking has also been learnt directly from tactile images either via recurrent [50], auto-encoder [15, 19] networks. Additionally, local track- ing has been explored with online factor-graph optimization [19, 20, 51]. The aforementioned ap- proaches are unimodal and rely on good pose initializations; MidasTouch is nonparametric and requires no such knowledge. Additionally, Kelestemur et al. [52] show category-level object pose estimation with a parallel jaw gripper.
Closely related is Tac2Pose [17, 18], which performs pose estimation of small-parts with the Gel- Slim [14]. It produces distribution of object poses learned in tactile simulation from contact shapes. Similarly, Gao et al. [53] estimates contact location on objects through vision- based touch and audio with small, discrete set of measurements. We differentiate our work in both context and approach: (i) we filter over long time horizons rather than single/multi touch predictions, (ii) tactile embed- dings are Iearned from local surface geometry rather than images, and (iii) we consider objects considerably larger than the robot finger.
Place recognition for touch: A compact representation for tactile images enables easy frame-to- frame Trame-to-mode tracking. Inspired by the computer vision community, popular interme-

---

diate is learned embedding from either RGB [15, 19] or binary contact masks [17. 18]. Realistic lactile simulators, like TACTO [32], enable training these models to the scale and generality of ar- bitrary real-world interactions. For example, Bauza et al. [18] learn object-specific embeddings to match 2D contact shapes.
For contact-rich manipulation, local 3D geometry is natural candidate, and iterative closest point (ICP) has shown promise for frame-to-frame tracking [20, 54]. For vision-based touch, 3D geometry is obtained either via photometric stereo [13, 55, 56], or image-to teee-ttmaaehhaappppp models [20, 31, 57- 59]. However, ICP is only suitable for local tracking and not global localization: it is sensitive to initialization and is intractable to scale as a sampling- based measurement model.
Succinctly, what is an accurate and efficient similarity metric for tactile geometry? With the ubiquity of LIDAR/RGB-D data, matching unordered point sets is vital for place recognition in the SLAM community [36, 37]. Following the seminal PointNet [60], Choy al. [61] later developed efficient and expressive sparse 3D convolution. This backbone was used to aggregate local point features to global point-cloud embeddings for LIDAR place-recognition [37]. We show tactile geometries can be compacted into codes with the same architecture, for easy frame-to-model queries.
Problem formulation
For ession-based tactile sensor dynamically sliding along known object's surface, our goal is to track the distribution 6D sensor pose x, E (3) in the object-centric frame. Such pose context useful for downstream planning and control, for instance manipulating an object in-hand [7]. The sensor affixed on a robot finger and we have access to the end-effector pose, but its relative position and orientation on the object's surface unknown. tactile image, and noisy sensor pose in the robot's reference-fraan frame, Z, { 2 P, SE(3)). }. For simplicity, we assume the object is stationary and sliding is single continuous contact interaction. The dimensions of the objects are much larger than the sensor's contact area. We assume an uninformative initial prior for x,, but later show the benefit of visual priors.
each timestep our measurements are
Global localization during sliding touch
MidasTouch comprises three distinct modules, illustrated in Figure I: tactile depth network (TDN: Section 4.1), tactile code network (TCN: Section 4.2), and particle filter (Section 4.3). At high-level, the TDN first converts tactile image I, into its local 3D geometry Pt via a learned ob- servation model. The 3D information is then condensed into a tactile code E, by the TCN through a sparse 3D convolution network. Finally, the downstream particle filter uses these learned codes in its measurement model, and outputs sensor pose distribution that evolves over time. Throughout this work, use 10 YCB objects with diverse geometries in our tests: sugar_box, tomato_sovv mug, power power_drill, scissors, adjustable, e_rench,, hammer, and baseball. In Appendix G, we further demonstrate the method small parts.
mustard_ botttr, bleach_cleamee
4.1 Tactile depth network (TDN)
The tactile depth network learns the inverse sensor model to recover local 3D geometry from a tactile image. We adapt dully-convolutions residual network [59, 62), and train in a supervised fashion to predict local heightmaps from tactile images.
Network and training: This network Re backbone, with a series of up- projection blocks. With the optical tactile sim- ulator TACTO [32], we render a large col- lection of DIGIT images with ground-truth heightmaps. The images are from simulated interaction with 40 YCB [24] object meshes, while holding out all test objects. The images rendered at 5000 different poses per object: randomizing for contact point, orientation, and indentation depth. For sim2real transfer,
30 gaunetry
DIGIT image
heightmap
image heightmap
Real-world DIGIT images from interactions in the YCB-Slide dataset (more in Appendix A). Our tactile depth network (Section 4.1) is trained simula- tion, and predicts 3D geometry given input tactile image.
Figure 2:

---

calibrate TACTO with real-world data from multiple independent DIGITs, and apply random light- ing augmentations (refer Appendix A). Per object, the split of vain-validation--esst is :000: :00: 500.
Image to 3D: At each timestep, we pass the 240 x 320 RGB tactile image I through the network to get heightmap H,. To remove non-contact areas we compute mask C, by depth thresholding, and get H H, C₁- Finally, the heightmap is reprojected to 3D via the camera's 's known perspective projection model H, + Pt. Evaluations on the test set show heightmap RMSE of 0.135 mm with respect to ground-trun Figure 2 shows examples of the TDN on real-world DIGIT interactions.
4.2 Tactile code network (TCN)
This network summarizes large, unordered point clouds of local geometry into a low-dimensional embedding space, code. If two sensor measurements are nearby in pose-space, they observe similar geometries and therefore, their codes will also be nearby in embedding-sppane space. However, the inverse is not necessarily true: measurements from two opposite corners of a cube may have similar codes, but their corresponding sensor poses are dissimilar. This is an inherent challenge of tactile localization, the so-called contact non-uniqueness highlighted by Bauza et al. [18]. In our work getting the most-likely modes of this distribution is sufficient since the downstream particle filter can then disambiguate them temporally.
An analog in the SLAM community is the loop closure problem from 3D data. LIDAR place recog- nition modules, such PointNetVLAD [36] and MinkLoc3D [37], use metric learning learn point cloud similarity. While tactile images differ from natural images, the geometries from LI- DAR and tactile sensors (normalized for scale) are similar. The MinkLoc3D architecture, based on MinkowskiNet [61]. performs state-of-the-art point cloud retrieval through sparse 3D convolutions.
The architecture comprises of three components: (i) Voxelization of P, into quantized sparse tensor P. = { (di, Di, 2,, 0/1)}, (ii) Feature pyramid network [63] for per-voxel local features P {ā Us. &, f. [ ). and (iii) Generalized-men mean pooling for point cloud embedding vector E, E _0 We use the pre-trained weights, learned from four comprehensive LIDAR datasets [37], and fine-tune them with TACTO data collected over the 40 YCB objects. This is trained with contrastive triplet loss, and we accumulate positive and negative local 3D geometries through pose-supervision.
Network and training:
Figure 3: Tactile codebook per object visualized as spectral colorspace map using t-SNE [64]. Each codebook comprises of 50k densely sampled poses with their corresponding 256-dimensional tactile code. Similar hues denote sensor poses that elicit similar tactile codes. We can clearly delineate local geometric features: edges (sugar_box). ridges (power_drile -nieerddrlll,. corners (scissors), and complex texture (baseball).
Once we have a code E,, we would like to efficiently compare this with dense set of contacts. Inspired by prior work [18, 26), we build a tactile codebook comprising of M = 50k randomized sensor poses on each object's ' mesh. We evenly sample these points and normals on the mesh with random orientations and indentations (refer Appendix A). We feed these poses into TACTO to generate dense set of tactile images for each object.
Tactile codebook:
We pass the generated images through the TDN + TCN to get codebook for the specific object o: C. = [p] Eimly) I. where !!!) are SE(3) sensor poses in codebook and E are correspond- ing tactile codes. Figure 3 shows t-SNE visualization of the codebooks, colorspace representation of local geometric similarity. We observe regions with similar geometries have identical hues and the traversal of our sensor between these geometries will give us valuable measurement signals.
We build a KD-Tree with 6-element vectors: {[peals a log (prol )] for nearest-neighbor search. Here, log(.) is the SO(3) logarithm map obtained via Theseus [65], and a = 0.01 is the rotation scaling factor. Thus, given candidate pose x, do not have to render and obtain
[m]]

---

its corresponding code, but instead just perform a pose-space lookup Co (x."). With memoization of code generation, we make getting codes for thousands of arbitrary pose particles tractable.
Single-touch localization: To understand the effectiveness of codes as a proxy for pose pre-
Nommlize tommlizedoo--25 top-25 poso ertor sumple
Randoni
diction, we conduct set of single-touch ex-
periments in simulation. This also provides idea of which objects are salient, and which are adversarial, highlighted in Figure 4. We ob- serve that we perform significantly better than random touches for all objects, and those with symmetric, regular structures exhibit long-tail errors. This shows that while single-touch has meaningful signal, it can be valuable to disambiguate these readings temporally with filtering framework. We compare against em- beddings from tactile images in Appendix B.
0.0
Figure 4: Pose-error for 50k single-touch queries YCB objects, ascending order (normalized with re- spect to random touch). For each query, we get the top- 25 highest scores from the tactile codebook Co. and com- pute their minimum pose-error with respect to ground- truth. We observe tools with salient geometries be easier to localize versus objects that exhibit symmetry.
4.3 Filtering over posed touch
The particle filter approximates the posterior distribution of the sensor pose as a set X, [(x]" w / where x!" are SE(3) sensor pose particles, uz! the predicted particle weights, and N, is number of particles. Rather than a unimodal Gaussian, this arbitrary distribution cap- tures the multi-modality of global localization. We propagate the posterior distribution based on the stream of tactile images and noisy sensor poses, emulating a conventional particle filter but with a learned measurement model. Additional implementation details can be found in Appendix C.
Initialization: The initial distribution Xo is sampled from a coarse prior, which can (optionally) be obtained from vision [26]. Estimating contact location from vision is inherently noisy and can be ambiguous due to object symmetries. In our experiments, find that even with very uninformative priors, our multi-modaa filter can prune out hypotheses and converge to the correct one.
~N(x Traas
We sample particles from 6D prior about the ground-trus pose x as X
rrot such that 30 = Maiag and 30 = 180" This weak prior scatters the particles across the object surface, and allows comparison across objects as function of their mesh diagonal length Maing In our results, we also show particle filter ablations with tighter pose priors. After sampling, we project the particles back onto the surface by querying their nearest neighbors from the tactile codebook.
We typically have access to noisy global estimates of end-effector pose, Pt, via robot kinematics. Our sensor odometry comes from the relative pose predictions Ap, (pt-1)" Pt- The motion model propagates particles X,-1 forward by sampling from state transition probability:
Motion model:
[8]
(xx:
x
Ap:) = [ C M(Apr Eap)
(1)
Additionally, odometry can also be from marker flow [66] or image-to-image tracking [19,20].
Measurement update: In this step, we update the particle weights based on how their tactile codes match the current measurement E,, as a function of cosine distance. At each timestep, we lookup the codebook for particle codes (Section 4.2) and perform matrix-vector multiplication:
w!! = p(z;
x
E.C.,( ||..|-11c.0>
softmax
(2)
These weights are scaled [0, 1] and represent how well the candidate particle poses match with the current measurement. These weights feed into the subsequent resampling step.
Particle resampling: We sample new of particles X, from the proposal distribution with probability proportional to their weights. Low-variance resampling [67] is popular solution that covers the sample set in systematic manner.
5

---

Hypothesis clustering: Alongside distribution of poses, downstream tasks may need distinct pose hypotheses. To achieve this, we hierarchically cluster particles in I3 using DB-SCAN [68]. We average the cluster positions and quaternions [69] to get a hypothesis set h4 {x....x?")
The YCB-Slide dataset
To evaluate Touch, and enable further research in tactile sensing, we introduce our YCB-Slide dataset (more in Appendix D). It comprises of DIGIT sliding interactions on the 10 YCB objects from Section 4. We envision this contribute towards efforts in tactile localization, mapping, object understanding, and learning dynamies models. We provide access DIGIT images, sensor poses, ground-truth mesh models, and ground-truth heightmaps contact masks (simulation only).
Eomple stmelatton Engaonice Injocintic
Figure 5: Example stiding trajectories from simulated and real trials on the 10 YCB objects. Overlaid in green are the local 3D geometries captured by the tactile sensor, and the contact poses RGB coordinate axes.
Simulated interactions: We simulate sliding across objects using TACTO, mimicking realistic interaction sequences. The pose sequences are geodesic-paths of fixed length L, = 0.5m, connecting random waypoints on the mesh. We corrupt sensor poses with zero zero-mean mean Gaussian noise tras 0.5mm, Zoo = 1 We record five trajectories per-object; 50 interactions in total. A representative sample of the trajectories, along with their accrued tactile geometries are shown in Figure 5.
Real-world interactions: In the real-world, handheld operation of the DIGIT. We keep each YCB object stationary with heavy- duty bench vise, and slide along the surface and record 60s trajectories at 30Hz. We use an OptiTraak system for timesynced sensor poses, with 8 cameras tracking the reflective markers. We affix six markers on the DIGIT and four on each test object. The canonical object pose is adjusted to agree with the ground- truth mesh models. Minor misalignment of sensor poses from human tories to lie on the object surface. We record five logs per- object, for a total of 50 sliding interactions. Our experimental setup and dataset are illustrated in Figures 5 and 6.
perform sliding experiments through
rectified by postprocessing the trajec-
Figure 6: Real- world sliding trials in the VCB-Slide dataset. Inset an example tactile image from the interactions, cap- turing the local geometry of the object.
Experimental results
Simulation: We evaluate MidasTouch over the 50 trajectories collected in Section 5. As the filter non-deterministic, each trajectory run 10 times for averaged results over 500 trials. Figure shows qualitative results for three representative trajectories. At each timestep, we visualize the pose distribution and plot the average particle RMSE with respect the ground-truth pose. We see convergence the most-likely pose hypothesis over the sliding interactions. Upon convergence, the hypothesis clustering step sets averaged pose for each cluster: visualized as the pose axes with uncertainty ellipses. We also highlight the current tactile image, surface geometry, and comparison with the tactile codebook. These trials are initialized with pose prior [otrams 01xx from Section 4.3.
Figure 8 [left] accumulates quantitative metrics over the 500 simulated trials. First, we show the final pose errors across all trials compared against the initial error. Overall, the averaged final pose errors are 0.74cm and 9.43%; the per-object errors roughly correlate the single- touch errors from Figure 4. While the error drops significantly for most trials, it fluctuates de- pending each trajectory's 's salient geometries (or lack thereof). Larger pose-crrors can be at-

---

Avg. IMMse trams, (mmm)
Aog. (0MS not. (Geg)
tomato, Sww.p .con
40
601
20
400
20
100
1000 Timestep
Avp. RMSE trans. (mm))
Ang: RMS ml (60g)
mustar bobttle
Go
so:
60
20
1000
Tlaestap
Ava, R30SE trans. Dani)
Aug. RMSS Fot. (deg)
bassebl11
20
20
20
3.0
100
500
1000 Thmestepp
100
500
1000
end
timesteps
Figure Snapshots simulated sliding results on three YCB objects. For each row: [top] the tactile images, local geometries, and heatmap of pose likelihood with respect to the tactile codchook, [bottom] pose distri- bution evolving over time, and converging the most- most-likely hypothesis after encountering salient geometries, [right] average translation and rotation RMSE of the distribution time with variance over 10 trials.
Final errors over 500 simulated trials
Trojectory-axengee trans, (mm)
Pose prior ablation over 500 trials
(deg)
100-
Initial
100*
Fizel RMSE trans. (mam)
l'inal ertor
Findl RMSE not. (deg)
80
All trojctories
8Dx
80-
80
60
Injectory averaged
40
40"
20-
20*
20
20
0.125
0.25
0.5
0.75
1.0
Pose prior ratio
Figure 8: [left] Boxplot of final trajectory error over 500 simulation trials. For each object, we plot the averaged initial and final RMSE for the particle set. We observe betler convergence for tools with salient geometries. opposed to symmetric objects. [right] Ablation over imitial pose uncertainty, show lower average trajectory error with better visual priors. With weaker initialization (3 1.0), outliers in pose-error are more prevalent.
tributed to tracking multiple modes that equally explain the same sliding sequence (please fer to supplementary video). We consider this benefit of our multi-modal framework, and is especially prevalent for symmetric objects like the sugar_box, mustard_bottlk and mug. We perform ablations over pose prior (Figure [right]), initializing each trial with uncertainty x [rrans Tra). This serves as a proxy for visual-priors; better initializations lead to fewer candidate modes. The full-stack operates, on average, the rate of 10Hz.
Final errors over 500 real trials
Initial enror
70
Pinal RMSE trams, (mm)
Final error
RMSE (deg)
50
Real-world: We run 500 similar trials for the real- world data from Section 5 with 3 = 0.5, which serves as a bound for a reasonable prior estimate available from vision. The real-world tactile heightmaps are noisier than simulation, thus tighter initialization pre- vents particle depletion. In Figure 10 we present three representative trajectories that show the hypotheses converge to the ground-truth pose. The accumulated
Figure Boxplot of final 500 real- world trials from the 50 YCB-Slide trajectories.

---

bleach cleanser
Aue ROUSE trans. (mmn
600
60
30
40
Timestop Immetop
Avg, TMMS trens (mm)
Aug. (00gg
BD
100
sugar bbx
80
400
Timestop
Aug. RMSSE tuns (mun)
60
100s
Ang: (dag)
scissors
73
40
50x
20
1000 Timestop
500
1000
1500
end
timesteps
Figure 10: Snapshots real-world sliding results on three YCB objects. each row: [top] tactile images, local geometries, and heatmap pose likelihood with respect to the tactile codebook, [bottom] pose distri- bution evolving over time, and converging to the most- likely hypothesis after encountering salient geometries, [right] average translation and rotation RMSE of the distribution over time with variance over 10 trials.
statistics in Figure 9 shows reduced final eггоr across all objects, except baseball. The averaged final pose enrors are higher than the simulation results: 1.97cm and 21.48%. Once again, we see tools and intricate objects are the easiest localize on while symmetric objects are the hardest. anomaly the baseball, on which we fail to localize in all trials. We attribute this lack of distinct geometry detected from real tactile images, effectively meaning we are trying localize on featureless sphere. These fallure modes are highlighted in Figure 11. We present best-hypothesis error metrics and further qualitative results in Appendix E.
7 Conclusion and limitations
In this work, we demonstrate finger-object global localiza- tion from posed tactile images. This online method outputs a pose-distribution on the object's surface that converges over time as the sensor traverses salient surface geometries. Specifically, our system is the first to learn tactile embed- dings from local 3D geometry, and disambiguate them with nonparametric particle filter. Our experiments demonstrate the surprising effectiveness of pairing learned tactile percep- tion with Monte-Carlo methods to resolve distribution am- biguities.
Object symmetries
Slow convergence Not enough geometry
Figure Failure modes the real- world: (i) it may be hard to converge to the true hypothesis for objects with sym- metries, (ii) slow convergence of the fil- lead to large pose uncertainty, (iii) lack discernible geometry can result drift from the true mode.
Limitations:
The current formulation limited to a mov- ing sensor relative fixed-pose object (or vice-versa). In future work. we wish to incorporate dynamic object in motion model through (i) visual measurements [26), and/or (ii) local in-hand tracking [20]. Our method currently does not work in scenarios where we lack ground-trutt -trut object models. Along with reconstructing objects [70], we would need to build the tactile codebook on-the-fly. For future in-hand manipulation tasks,
also necessary to scale Mi- das' doschucc to multi-contact configurations. Our on-surface assumption leads to poor behavior when we break contact with the object, and the effects of shear on the tactile image sequence have been ignored [71]. Object compositionality doesn't it hold for deformable or articulated objects, mod- eling these properties is interesting future direction [72]. With differentiable filter [73] we fine-tune the system end-to-end for more robust real-world performance.

---

Acknowledgments
We thank Wenzhen Yuan, Ming-Fann Fang Chang, Wei Dong, Maria Bauza Villalonga, and Antonia Bronars for insightful discussions. We are grateful towards the CMU AI Maker Space and Greg Armstrong facilitating the collection of the YCB-Slide dataset. The authors acknowledge funding from Meta AI, and this work was partially carried out while Sudharshan Suresh interned at Meta Al.
References
[1] J. Heller. Catch-22: novel, volume 4. Simon and Schuster, 1961.
[2] Shi, J. Z. Woodruff, P. B. Umbanhowar, and M. Lynch. Dynamic in-hand sliding manip- ulation. IEEE Trans. Robotics (TRO), 33(4))778-795, 2017.
[3] N. Chavan-Dafle, R. Holladay, and A. Rodriguez. In-hand manipulation via motion cones. Proc. Robotics: Science and Systems (RSS), 2018.
[4] J. Shi, H. Weng, and K. M. Lynch. In-hand sliding regrasp with spring-sliding compliance and external constraints. IEEE Access, 8:88729-88744, 2020.
[5] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson. Cable manipulation with tactile-reactive gripper. Intl. J. of Robotics Research (IJRR), 40(12-14):1385-14444 1401, 2021.
[6] J. Kerr, H. Huang, A. Wilcox, R. Hoque, J. Ichnowski, R. Calandra, and K. Goldberg. Learn- ing self-supervised representations from vision and touch for active sliding perception of de- formable surfaces. arXiv preprint arXiv:2209. 13042, 2022.
[7] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. Intl. J. of Robotics Research (IJRR), 39(1):3-20) 2020.
[8] Chen, Xu, and P. Agrawal. system for general in-hand object re-orientation. In Proc. Conf. on Robot Learning (CoRL), pages 297-307. PMLR, 2022.
[9] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk,
K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. DeXtreme: Transfer of agile in- m-hand manipulation from simulation to reality. arXiv, 2022.
[10] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[11] B. Sundaralingam and T. Hermans. Geometric in-hand regrasp planning: Alternating opti- mization of finger gaits and in in-grasp manipulation. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 231-20 231-238. IEEE, 2018.
[12] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. In Proc. JEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 2386-2393. IEEE, 2019.
[13] W. Yuan, S. Dong, and E. H. Adelson. GelSight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[14] E. Donlon, S. Dong, M. Liu, Li, E. Adelson, and A. Rodriguez. GelSlim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In Proc. JEEE/RSJ Inil. Conf. on Intelli- gent Robots and Systems (IROS), pages 1927-1934, IEEE, 2018.
[15] M. Lambeta, P.W. Chou, S. Tian, Yang, Maloon, R. Most, Stroud, R. Santos, A. Byagowi, G. Kammerer, al. DIGIT: A novel design for low-cost compact high- resolution tactile sensor with application in-hand manipulation. IEEE Robotics and Au- tomation Letters (RA-L), 5(3):3838-33) 2020.

---

[16] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson. Localization and manipulation of small parts using GelSight tactile sensing. In Proc. IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 3988-3993. IEEE, 2014.
[17] M. Bauza, E. Valls, B. Lim, T. Sechopoulos, and A. Rodriguez. Tactile object pose estimation from the first touch with geometric contact rendering. In Proc. Conf. on Robot Learning, CoRL, 2020.
[18] M. Bauza, A. Bronars, and A. Rodriguez. Tac2Pose: Tactile object pose estimation from the first touch. arXiv preprint arXiv:2204. 1701, 2022.
[19] P. Sodhi, M. Kaess, M. Mukadam, and S. Anderson. Learning tactile models for factor graph- based estimation. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 13686- 13692. IEEE, 2021.
[20] P. Sodhi, M. Kaess, M. Mukadam, and S. Anderson. Patchgraph: In-hand tactile tracking with learned surface normals. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), 2022.
[21] J. Hawkins, M. Lewis, M. Klukas, S. Purdy, and S. Ahmad. A framework for intelligence and cortical function based grid cells in the neocortex. Frontiers in neural circuits, 12:121, 2019.
[22] J. Lederman and R. L. Klatzky. Hand movements: A window into haptic object recognition. Cognitive psychology, 19(3):342-368, 1987.
[23] Dellaert, D. Fox, W. Burgard, and S. Thrun. Monte Carlo localization for mobile robots. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), volume 2, pages 1322-1328 IEEE, 1999.
[24] B. Calli, A. Singh, Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Yale-CMU--Berelle dataset for robotic manipulation research. Intl. J. of Roboties Research (IJRR), 36(3);261-268, 2017.
[25] L. Downs, A. Francis, N. Koenig, Kinman, Hickman, K. Reymann, B. McHugh, and
V. Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items.
Proc. IEEE Inil. Conf. Robotics and Automation (ICRA), 2022.
[26] X. Deng, A. Mousavian, Y. Xiang, Xia, T. Bretl, and D. Fox. PoseRBPF: A Rao- Blackwellized particle filter for 6D object pose tracking. IEEE Trans. Robotics (TRO), 1328):322-11422 2021.
[27] A. Yamaguchi and C. G. Atkeson. Combining finger vision and optical tactile sensing: Reduc- ing and handling errors while cutting vegetables. In Proc. IEEE-RAS Intl. Conf. Humanoid Robots (Humanoids), pages 1045-1051. IEEE, 2016.
[28] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, Rossiter, and N. F. Lepora. The TacTip family: Soft optical tactile sensors with 3D 3D-printed biomimetic morphologies. Soft robotics, 5(2):216-227, 2018.
[29] A. Alspach, K. Hashimoto, N. Kuppuswamy, and Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor for manipulation. In Proc. IEEE Intl. Conf. on Soft Robotics (RoboSoft), pages 597-604. IEEE, 2019.
[30] A. Padmanabha, F. Ebert, S. Tian, R. Calandra, C. Finn, and S. Levine. OmniTact: A multi- directional high-resolution touch sensor. In Proc. IEEE Intl. Conf. Robotics and Automation (ICRA), pages 618-624. IEEE, 2020.
[31] S. Wang, Y. She, B. Romero, and E. Adelson. GelSight Wedge: Measuring high-resolution 3D contact geometry with compact robot finger. In Proc. IEEE Intl. Conf. Robotics and Automation (ICRA). IEEE, 2021.
10

---

[32] S. Wang, M. M. Lambeta, P.-.. Chou, and R. Calandra. TACTO: A fast, flexible, and open- source simulator for high-resolution rision-based tactile sensors. IEEE Robotics and Automa- tion Letters (RA-L), 2022.
[33] A. Agarwal, T. Man, and W. Yuan. Simulation of vision-based tactile sensors using physics based rendering. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 1-7. IEEE, 2021.
[34] Z. Si and W. Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters (RA-I.), 2022.
[35] C. Choy, J. Park, and V. Koltun. Fully convolutional geometric features. In Proc. Inil. Conf. on Computer Vision (ICCV), pages 3958-8966, 2019.
[36] M. A. Uy and G. H. Lee. PointNetVLAA: Deep point cloud based retrieval for large-scale place recognition. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 4470-4479, 2018.
[37] Komorowski. MinkLoc3D: Point cloud based large-scale place recognition. In Proc. Winter Conf. on Applications of Computer Vision (WACV), pages 1790-1799, 2021.
[38] C. Corcoran and R. Platt. Tracking object pose and shape during robot manipulation based on tactile information. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), volume 2. Citeseer, 2010.
[39] A. Petrovskaya and O. Khatib. Global localization of objects via touch. IEEE Trans. on Robotics (TRO), 276))569-585,, 2011.
[40] L. Zhang and J. C. Trinkle. The application of particle filtering to grasping acquisition with visual occlusion and tactile sensing. In 2012 IEEE International Conference on Robotics and Automation, pages
IEEE, 2012.
[41] M. Chalon, Reinecke, and M. Pfanne. Online in-hand object localization. In Proc. IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 2977-2984. IEEE, 2013.
[42] B. Saund, S. Chen, and Simmons. Touch based localization of parts for high precision manufacturing. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 378-385. IEEE, 2017.
[43] M. C. Koval, M. Klingensmith, S. S. Srinivasa. N. S. Pollard, and M. Kaess. The manifold particle filter for state estimation on high-dimensional implicit manifolds. In Proc. IEEE Inil. Conf. on Robotics and Automation (ICRA), pages 4673-4680, IEEE, 2017.
[44] S. Thrun. Particle filters robotics. In UAI, volume 2, pages 511-518. Citeseer, 2002.
[45] Z. Pezzementi, C. Reyda, and G. D. Hager. Object mapping, recognition, and localization from tactile geometry. In 2011 IEEE International Conference on Robotics and Automation, pages 5942-5948. IEEE, 2011.
[46] S. Luo, W. Mou, K. Althoefer, and H. Liu. Localizing the object contact through matching tactile features with visual map. In Proc. IEEE Inil. Conf. on Robotics and Automation (ICRA), pages 3903-3908, IEEE, 2015.
[47] Bimbo, S. Luo, K. Althoefer, and H. Liu. In-hand object pose estimation using covariance- based tactile to geometry matching. IEEE Robotics and Automation Letters (RA-L), I(I); 970-577, 2016.
[48] K.-T. Yu and A. Rodriguez. Realtime state estimation with tactile and visual sensing, appli- cation to planar manipulation. In Proc. IEEE Intl. Conf. Robotics and Automation (ICRA), pages 7778-7785 IEEE, 2018.

---

[49] A. N. Chaudhury, T. Man, W. Yuan, and C. G. Atkeson. Using collocated vision and tactile sensors for visual servoing and localization. IEEE Robotics and Automation Letters (RA-L), (2):347-33434, 2022.
[50] S. Dong and A. Rodriguez. Tactile-based insertion for dense box-packing. Proc. JEEE/RSI Inil. Conf. on Intelligent Robots and Systems (IROS), pages 7953-7960. IEBE, 2019.
[51] S. Kim and A. Rodriguez. Active extrinsic contact sensing: Application to general peg-in-hole insertion. In Proc. IEEF Intl. Conf. on Robotics and Automation (ICRA). IEEE, 2022.
[52] T. Kelestemur, R. Platt, and T. Padir. Tactile pose estimation and policy learning for unknown object manipulation. arXiv preprint arXiv:2203.10685 2022.
[53] R. Gao, Si, Y--Y. Chang, S. Clarke, Bohg, L. Fei-Fei, W. Yuan, and J. Wu. ObjectFolder 2.0: A multisensory object dataset for Sim2Real transfer. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
[54] N. Kuppuswamy, A. Castro, C. Shillips-Graffiin A. Alspach, and R. Tedrake. Fast model- based contact patch and pose estimation for highly deformable dense-geometry tactile sensors. IEEE Robotics and Automation Letters (RA-L), 1(2):1811-1818, 2019.
[55] M. K. Johnson and E. H. Adelson. Retrographic sensing for the measurement of surface texture and shape. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 1070-1077, 2009. doi: 10. 111.///P..209955066653
[56] M. K. Johnson, F. Cole, A. Raj, and E. H. Adelson. Microgeometry capture using an elas- tomeric sensor. ACM Trans. Graph., 30(4), July 2011. ISSN 0730-0301.
[57] M. Bauza, O. Canal, and A. Rodriguez. Tactile mapping and localization from high-resolution tactile imprints. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 3811 3817. IEEE, 2019.
[58] R. Ambrus, V. Guizilini, N. Kuppuswamy, A. Beaulieu, A. Gaidon, and A. Alspach. Monoc- ular depth estimation for soft visuotactile sensors. In Proc. IEEE Intl. Conf. on Soft Robotics (RoboSoft), 2021.
[59] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess. ShapeMap 3-D: Efficient shape mapping through dense touch and vision. Proc. IEEE Intl. Conf. on Roboties and Automation (ICRA), Philadelphia, PA, USA, May 2022.
[60] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning point sets for 3D classi- fication and segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 552-660,22 2017.
[61] C. Choy, J. Gwak, and S. Savarese. 4D spatio-temporal ConvNets: Minkowski convolutional neural networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3075-3084, 2019.
[62] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. Deeper depth prediction with fully convolutional residual networks. In Proc. Intl. Conf. on 3D Vision (3DV), pages 239-248. IEEE, 2016.
[63] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid net- works for object detection. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 21 2117-2125, 2017.
[64] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
12

---

[65] L. Pineda, T. Fan, M. Monge, S. Venkataraman, P. Sodhi, R. T. Chen, J. Ortiz, D. DeTone, A. Wang, S. Anderson, J. Dong, B. Amos, and M. Mukadam. Theseus: A Library for Differ- entiable Nonlinear Optimization. Advances in Neural Information Processing Systems, 2022.
[66] W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson. Measurement of shear and slip with GelSight tactile sensor. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 304-311, IEEE, 2015.
[67] S. Thrun, Probabilistic robotics. Communications of the ACM, 45(3): 2002.
[68] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu. DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Transactions on Database Systems (TODS), 42(3):1-21, 2017.
[69] F. L.. Markley, Y. Cheng, J. L. Crassidis, and Y. Oshman. Averaging quaternions. Journal of Guidance, Control, and Dynamics, 30(4):1193-1197, 2007.
[70] Suresh, M. Bauza, K.-T. Yu, J. G. Mangelson, A. Rodriguez, and M. Kaess. Tactile SLAM: Real-time inference of shape and pose from planar pushing. In Proc. IEEE Intl. Conf. Robotics and Automation (ICRA), May 2021.
[71] K. Aquilina, D. A. Barton, and F. Lepora. Shear-invariant sliding contact perception with a soft tactile sensor. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 4283-4289. IEEE, 2019.
[72] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. VIRDO: Visio-tactile implicit representations of deformable objects. In Proc. IEEE Intl. Conf. on Robotics and Automation (ICRA). IEEE, 2022.
[73] R. Jonschkowski, D. Rastogi, and O. Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. In Proc. Robotics: Science and Systems (RSS), 2018.
[74] McMaster-Carr, htpp://www..emmsterr.om/// Accessed: 2022-08-25.
13


Reviews:
[{"review_id": "Ol2M3krSrt9", "reviewer": "Reviewer_tqe3", "summary": "The paper proposes an approach for global localization of an object from tactile sensing and noisy pose of the tactile sensor in the global frame (similar to odometry in SLAM). The paper sets up this problem in a similar setting to localization in a known map from the SLAM community. \n\nThe paper converts RGB images from DIGIT tactile sensor to depth using a NN, followed by learning a latent feature space (tactile code) from the estimated depth. This tactile code is used to compare between tactile readings. A particle filter is then used to infer the pose of the tactile sensor w.r.t. the object based on the tactile code at different poses. Offline tactile code is generated from the known object mesh to use in the particle filter. The proposed approach requires an object mesh to work.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written with extensive validation of the approach in sim and the real world. I appreciate the creation of a real world dataset which will enable other methods to benchmark or use without requiring expensive tracking systems. The paper also does a good job in explaining limitations and also experimentally showing these limitations. \n\nThe paper is in very good shape. I have some feedback to help improve it which I list below:\n\n1. Recording and plotting the force applied with the DIGIT sensor during data collection would help understand the relationship between large contact area vs small contact area. One experiment would be to run the same trajectory but with two different force profiles. I suspect that the orientation error will become worse as the force applied becomes lower. \n\n2. Why is there a very high orientation error on the real world dataset? Is this because of not sufficient contact area during data collection? Adding a plot of the contact patch area would help in analyzing this question.\n\n3. Why was the pose prior for the object chosen to be in a narrower range for the real world dataset? (line 260). Describing why this was chosen in the paper would be helpful. \n\n4. When I read this paper, I started thinking on how this could be applied to a real world manipulation demo. One idea would be to show localization on a transparent object (where vision based pose estimation would fail). Showing an example with a transparent object would help readers understand the need for tactile localization (in contrast to vision).\n\n5. Does passing the RGB image from DIGIT to the tactile code network help disambiguate geometrically similar but visually different object regions? E.g., a Pringles can or a mustard bottle (front has a different label compared to back). I am curious if this was attempted.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written with thorough experiments to validate the proposed approach. The paper also creates a real world dataset for use by the community.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Provide explanation to the below concerns (from strengths/weakness section of the review):\n\n2. Why is there a very high orientation error on the real world dataset? Is this because of not sufficient contact area during data collection? Adding a plot of the contact patch area would help in analyzing this question.\n\n3. Why was the pose prior for the object chosen to be in a narrower range for the real world dataset? (line 260). Describing why this was chosen in the paper would be helpful. \n"}}, {"review_id": "Lc3HkQlDwDB", "reviewer": "Reviewer_Cxtg", "summary": "This paper presents a tactile perception system that mimics LIDAR for place recognition, where an off-the-shelf tactile sensor is used as the LIDAR, placed on the surface of YCB objects as the environment, then collects data through a sliding motion to determine sensor localization. The authors intend to release the dataset, but no link has been provided yet. The attached code is not runnable, as explained by the authors. The overall idea is exciting and relatively straightforward once you make the comparison with LIDAR. \n\nThe major contribution would be the idea of connecting tactile perception with LIDAR. Although its specific usage remains unclear, at least not well-justified in the current paper. \n\nReleasing such a dataset would be a contribution (although not yet released in the paper). \n\nThe learning part is no surprise, but I expect to see a direct transfer of the SLAM method applied in this dataset to see if the result or method is transferrable or to what extent. ", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nThe analogy from SLAM to Touch is interesting. The way data is collected is straightforward, and the data collected from the simulated and actual environments are solid and concrete. \n\nWeakness:\n-\tNo data has been released yet.\n-\tUnclear how the data could be used. Objects are much smaller than large scene environments. On the practical side, it is challenging to generate a scene map using one sensor (or even multiple sensors) with one or more robots exploring the scene with uncertainty. But objects are usually much smaller and, therefore, could be digitally captured using many vision-based methods in one shot. So what would be the usage scenarios to justify the proposed data?\n-\tUnclear how robots could be involved. Again, this is not well justified throughout this paper. \n-\tSince the object data is already known, what if we take a square box and slide it over the object’s point cloud to collect the same data, then add noise to it to make it real? Wouldn’t it be much easier?\n-\tThis paper only discussed the geometric features through the point cloud for tactile sensors. But from a robotics perspective, we are more interested in the physical interactions on the touch interface with respect to the object, including data regarding force and torque. Unfortunately, nothing is reported in this paper, making the data less useful for robotic applications. As that seems to be the crown jewel with tactile data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work is pretty detailed, and the idea is interesting, but its relevance to robotics is not well justified. It is more related to computer vision based on the current content. Still, with tactile data, further emphasis on the physical interactions for robotic manipulation should be explained, which is not presented in the current paper or data. Unlike SLAM, where the scene is much more extensive than common sensing technologies, sensors such as LIDAR are used to generate maps for localization or provide localized details for navigation. But objects are much smaller in size (than most common sensors). The use of tactile information (in terms of robotics) falls mainly in understanding the physical properties that are not easily accessible to common sensors or the localized interaction force and torque to provide further dynamics for manipulation. I find the data interesting but did not see sufficient evidence to make such tactile data or learning applicable in robotics based on the current content.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Further justification on how the data collected could be used in a meaningful way for robotic applications or solving robotic-related problems would be a vital issue to be addressed.\n- Any force-related information collected? If so, how would such information be used for robot learning? \n- Can any existing method used in SLAM be transferrable to the presented data set for a similar purpose? To what extent? Why it can or why it cannot be transferred?\n- The sliding motion used to collect data is conducted by hand. How much influence of this procedure would influence the usage of the collected data for robotics? If one attaches the same tactile sensor to slide on objects? Can it be done in the same way? How does one regulate the contact force or sensor pose during sliding?\n- During manual data collection, as presented by the author, some of the objects are not as rigid or firm, so manual contact through the sensor may deform the objects (I suspect). How much influence would it influence the quality of the collected data?\n- The related data is not yet published during the review of this paper.\n- 4 pages of references are too many for an 8-page conference paper.\n"}}, {"review_id": "D3JcUq3WU3z", "reviewer": "Reviewer_xg8x", "summary": "The work develops an online tactile perception system to build global context for relative pose tracking that tracks evolving pose distribution of vision based touch sensor. For this, they propose three distinct modules (depth network, code network, and particle filter), that converts the tactile image into its 3D local geometry, which is further condensed into a code / embedding via a convolutional network; these codes are used to output sensor pose distribution that evolves over time to build global localisation. The work performs experiments over 10 distinct YCB objects. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths : the work builds (1) a novel approach to global localisation for tactile sensing via long-horizon online particle filtering, (2) does not rely on pose initialisations and (3) uses local surface geometry instead of images to learn tactile embeddings \n\nweaknesses : (1) does not account for dynamic pose tracking ( sensor relative to object ) (2)  most of the experiments focus on large everyday objects; the work does not include enough experiments on relatively smaller-sized objects so it is insufficient to conclude whether the system works on varied sized objects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I read the paper thoroughly, and used my best judgement in assessing the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I would like to see experiments on varied-sized objects (example : tennis ball, marble, computer mouse, orange). Limited real world experiments included in the paper.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "4bep3Dskk2m", "reviewer": "Reviewer_aApi", "summary": "This paper uses high-resolution tactile sensing to estimate the relative sensor-object pose of known 3D objects with object models. The pose estimation is via estimating local object geometry under contact, which updates a distribution of object poses. Sliding over the object results in a successively more localized distribution of object poses. This has a connection with haptic exploratory procedures that humans are known to use to make decisions about object properties. Methods to estimate object pose are trained in simulation, and demonstrated with hand-held motion of a tactile sensor across real objects.    ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Complete pipeline from lcoal representation of tactile data inference of object pose based on object model. Code, models, data are all openly released.\n- Solution to a demanding problem that has long been a challenge to the field.\n- Good combination of simulation-based and real data-based methods to address the problem. Good combination of deep learning and conventional sampling methods.\n- Clear how this work opens up new capabilites that will be of benefit to the field - e.g. estimating object pose could be a prerequisite to object manipulation.\n\nWeaknesses\nOnly minor comments, as I think the paper is basically fine as it is.\n- I was a bit unsure about how and when you would apply the motion model. For human held motion, presumably this is due to noise in vision-based odometry? (e.g. in fig 6 you can see the attached markers for estimating pose). Robot arms can have very precise reporting of end effector pose, so would a motion model be necessary then?\n- Any thoughts about the tactile sensor? You use a DIGIT, but is this well suited to sliding motion, as past work has mainly considered dabbing motion with the sensor?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Strong well-written paper making a substantial contribution to the field.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see above", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)