Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes an approach for global localization of an object from tactile sensing and noisy pose of the tactile sensor in the global frame (similar to odometry in SLAM). The paper sets up this problem in a similar setting to localization in a known map from the SLAM community. 

The paper converts RGB images from DIGIT tactile sensor to depth using a NN, followed by learning a latent feature space (tactile code) from the estimated depth. This tactile code is used to compare between tactile readings. A particle filter is then used to infer the pose of the tactile sensor w.r.t. the object based on the tactile code at different poses. Offline tactile code is generated from the known object mesh to use in the particle filter. The proposed approach requires an object mesh to work.
strengths_and_weaknesses: The paper is well written with extensive validation of the approach in sim and the real world. I appreciate the creation of a real world dataset which will enable other methods to benchmark or use without requiring expensive tracking systems. The paper also does a good job in explaining limitations and also experimentally showing these limitations. 

The paper is in very good shape. I have some feedback to help improve it which I list below:

1. Recording and plotting the force applied with the DIGIT sensor during data collection would help understand the relationship between large contact area vs small contact area. One experiment would be to run the same trajectory but with two different force profiles. I suspect that the orientation error will become worse as the force applied becomes lower. 

2. Why is there a very high orientation error on the real world dataset? Is this because of not sufficient contact area during data collection? Adding a plot of the contact patch area would help in analyzing this question.

3. Why was the pose prior for the object chosen to be in a narrower range for the real world dataset? (line 260). Describing why this was chosen in the paper would be helpful. 

4. When I read this paper, I started thinking on how this could be applied to a real world manipulation demo. One idea would be to show localization on a transparent object (where vision based pose estimation would fail). Showing an example with a transparent object would help readers understand the need for tactile localization (in contrast to vision).

5. Does passing the RGB image from DIGIT to the tactile code network help disambiguate geometrically similar but visually different object regions? E.g., a Pringles can or a mustard bottle (front has a different label compared to back). I am curious if this was attempted.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written with thorough experiments to validate the proposed approach. The paper also creates a real world dataset for use by the community.
issues: Provide explanation to the below concerns (from strengths/weakness section of the review):

2. Why is there a very high orientation error on the real world dataset? Is this because of not sufficient contact area during data collection? Adding a plot of the contact patch area would help in analyzing this question.

3. Why was the pose prior for the object chosen to be in a narrower range for the real world dataset? (line 260). Describing why this was chosen in the paper would be helpful.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work develops an online tactile perception system to build global context for relative pose tracking that tracks evolving pose distribution of vision based touch sensor. For this, they propose three distinct modules (depth network, code network, and particle filter), that converts the tactile image into its 3D local geometry, which is further condensed into a code / embedding via a convolutional network; these codes are used to output sensor pose distribution that evolves over time to build global localisation. The work performs experiments over 10 distinct YCB objects. 
strengths_and_weaknesses: strengths : the work builds (1) a novel approach to global localisation for tactile sensing via long-horizon online particle filtering, (2) does not rely on pose initialisations and (3) uses local surface geometry instead of images to learn tactile embeddings 

weaknesses : (1) does not account for dynamic pose tracking ( sensor relative to object ) (2)  most of the experiments focus on large everyday objects; the work does not include enough experiments on relatively smaller-sized objects so it is insufficient to conclude whether the system works on varied sized objects.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I read the paper thoroughly, and used my best judgement in assessing the paper.
issues: I would like to see experiments on varied-sized objects (example : tennis ball, marble, computer mouse, orange). Limited real world experiments included in the paper.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a tactile perception system that mimics LIDAR for place recognition, where an off-the-shelf tactile sensor is used as the LIDAR, placed on the surface of YCB objects as the environment, then collects data through a sliding motion to determine sensor localization. The authors intend to release the dataset, but no link has been provided yet. The attached code is not runnable, as explained by the authors. The overall idea is exciting and relatively straightforward once you make the comparison with LIDAR. 

The major contribution would be the idea of connecting tactile perception with LIDAR. Although its specific usage remains unclear, at least not well-justified in the current paper. 

Releasing such a dataset would be a contribution (although not yet released in the paper). 

The learning part is no surprise, but I expect to see a direct transfer of the SLAM method applied in this dataset to see if the result or method is transferrable or to what extent. 
strengths_and_weaknesses: Strength: 
The analogy from SLAM to Touch is interesting. The way data is collected is straightforward, and the data collected from the simulated and actual environments are solid and concrete. 

Weakness:
-	No data has been released yet.
-	Unclear how the data could be used. Objects are much smaller than large scene environments. On the practical side, it is challenging to generate a scene map using one sensor (or even multiple sensors) with one or more robots exploring the scene with uncertainty. But objects are usually much smaller and, therefore, could be digitally captured using many vision-based methods in one shot. So what would be the usage scenarios to justify the proposed data?
-	Unclear how robots could be involved. Again, this is not well justified throughout this paper. 
-	Since the object data is already known, what if we take a square box and slide it over the object’s point cloud to collect the same data, then add noise to it to make it real? Wouldn’t it be much easier?
-	This paper only discussed the geometric features through the point cloud for tactile sensors. But from a robotics perspective, we are more interested in the physical interactions on the touch interface with respect to the object, including data regarding force and torque. Unfortunately, nothing is reported in this paper, making the data less useful for robotic applications. As that seems to be the crown jewel with tactile data.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The work is pretty detailed, and the idea is interesting, but its relevance to robotics is not well justified. It is more related to computer vision based on the current content. Still, with tactile data, further emphasis on the physical interactions for robotic manipulation should be explained, which is not presented in the current paper or data. Unlike SLAM, where the scene is much more extensive than common sensing technologies, sensors such as LIDAR are used to generate maps for localization or provide localized details for navigation. But objects are much smaller in size (than most common sensors). The use of tactile information (in terms of robotics) falls mainly in understanding the physical properties that are not easily accessible to common sensors or the localized interaction force and torque to provide further dynamics for manipulation. I find the data interesting but did not see sufficient evidence to make such tactile data or learning applicable in robotics based on the current content.
issues: - Further justification on how the data collected could be used in a meaningful way for robotic applications or solving robotic-related problems would be a vital issue to be addressed.
- Any force-related information collected? If so, how would such information be used for robot learning? 
- Can any existing method used in SLAM be transferrable to the presented data set for a similar purpose? To what extent? Why it can or why it cannot be transferred?
- The sliding motion used to collect data is conducted by hand. How much influence of this procedure would influence the usage of the collected data for robotics? If one attaches the same tactile sensor to slide on objects? Can it be done in the same way? How does one regulate the contact force or sensor pose during sliding?
- During manual data collection, as presented by the author, some of the objects are not as rigid or firm, so manual contact through the sensor may deform the objects (I suspect). How much influence would it influence the quality of the collected data?
- The related data is not yet published during the review of this paper.
- 4 pages of references are too many for an 8-page conference paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper uses high-resolution tactile sensing to estimate the relative sensor-object pose of known 3D objects with object models. The pose estimation is via estimating local object geometry under contact, which updates a distribution of object poses. Sliding over the object results in a successively more localized distribution of object poses. This has a connection with haptic exploratory procedures that humans are known to use to make decisions about object properties. Methods to estimate object pose are trained in simulation, and demonstrated with hand-held motion of a tactile sensor across real objects.    
strengths_and_weaknesses: Strengths
- Complete pipeline from lcoal representation of tactile data inference of object pose based on object model. Code, models, data are all openly released.
- Solution to a demanding problem that has long been a challenge to the field.
- Good combination of simulation-based and real data-based methods to address the problem. Good combination of deep learning and conventional sampling methods.
- Clear how this work opens up new capabilites that will be of benefit to the field - e.g. estimating object pose could be a prerequisite to object manipulation.

Weaknesses
Only minor comments, as I think the paper is basically fine as it is.
- I was a bit unsure about how and when you would apply the motion model. For human held motion, presumably this is due to noise in vision-based odometry? (e.g. in fig 6 you can see the attached markers for estimating pose). Robot arms can have very precise reporting of end effector pose, so would a motion model be necessary then?
- Any thoughts about the tactile sensor? You use a DIGIT, but is this well suited to sliding motion, as past work has mainly considered dabbing motion with the sensor?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Strong well-written paper making a substantial contribution to the field.
issues: Please see above
review_title: null

Meta-review: metareview: The paper presents a tactile perception system with real DIGIT sensors to estimate relative sensor-object pose of known 3D objects with object models. Reviewers agree that the proposed approach is interesting, limitations are well articulated, and that the dataset will be a valuable contribution to the community. Reviewers also raise several questions that need additional clarification. Authors are encouraged to respond to reviewers' comments and questions.

Not releasing the dataset at the time of submission (which can be due to a myriad of reasons e.g. dataset size, data release policies, etc.) is not grounds for rejection. However, authors are encouraged clarify the relevance to robotics.

Update:

Reviewers appreciate the authors' detailed responses and modifications. The work tackles an important problem in tactile perception (e.g. for manipulation), provides an accessible dataset to accelerate tactile learning research, and extensive real world experiments and analysis – all of which are contributions that are highly relevant to the robotics community. Including the revisions, the manuscript is overall well polished, and should make for an excellent oral presentation.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)