Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this work, a new dataset for multimodal visuo-tactile learning called Touch and Go is proposed,  in which human data collectors probe objects in natural environments with tactile sensors, while recording egocentric video.
The objects and scenes in this dataset are significantly more diverse than prior efforts. Experiments on three kinds of tasks have been conducted to demonstrate this dataset’s effectiveness :-1) self-supervised visuo-tactile feature learning, -2) tactile-driven image stylization,
-3) predicting future frames of a tactile signal from visuo-tactile inputs.  
strengths: ---1) The data are collected by human in an “approximately egocentric” manner,  which is thought to have higher diversity than those data collected by robots.
---2) Compared with previous efforts, this dataset has the largest scale in terms of the number of categories of objects and scences. It also contains the largest number of touches.
---3) Extensive experiments have been done, and the results show that this dataset is effective on different downstream tasks.
weaknesses: 1) the timings of touch onsets are labeled by machine instead of human, which may cause error of labeling.
2) there is no comparison with other visuo-tactile datasets, so it's no clear what benefits this dataset can bring to the downstream tasks. 
correctness: The dataset is constructed in a sound way. And expriements cover a wide range of tasks. the results show the effectiveness of this dataset. but the comparison with other visuo-tactile datasets are missed.
clarity: As far as I am concerned,this paper can clearly deliver its main controbution.
relation to prior work: yes. The objects and scenes in this dataset are significantly more diverse than prior efforts, making the data well-suited to tasks that involve
understanding material properties and physical interactions in the wild.
additional feedback: None
review_title: An large dataset for multimodal visuo-tactile learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper provide a large visual-tactile dataset with GelSight tactile sensor and camera, containing diverse senarios and objects.Experimental analysis is carried out on self-supervised feature learning, tactile-driven image stylization and future tactile image prediction.
strengths: 1. The scale of the proposed dataset is larger than previous ones and the diversity of objects and senarios is enhanced.
2. Various applications are presented on the proposed dataset, including a novel tactile-driven image stylization task.
weaknesses: 1. The functionality of the dataset remains undetermined since the results are not compared with those of other visual-tactile datasets (e.g. for Robotic grasping only ImageNet but no other visual-tactile datasets are compared).
2. Although interesting, the motivation of the proposed tactile-driven image stylization is not clearly stated, and the qualitative results seem trivial.
3. Benchmarking on the proposed dataset is inadequate. For touch prediction only one model is evaluated.
correctness: The claims appear to be correct.
clarity: The paper is well written.
relation to prior work: Prior works are compared with the proposed dataset and the difference in scale are highlighted.
additional feedback: No additional feedback.
review_title: Review for Touch and Go

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: To address the lack of sight-touch correlation data, the authors proposed a multi-modal visuo-tactile dataset.
Instead of being robot-centric like prior efforts, with wearable tactile sensors and egocentric camera, human-centric data involving diverse objects and scenes are collected.
The dataset empowers multiple tasks, including tactile feature prediction, newly proposed tactile-driven image stylization, and future touch prediction.

strengths: 1. Given the drawbacks of robot-centric and simulation-based methodology for visuo-tactile data collection, the authors proposed to collect human-centric data using tactile sensors and ego-centric camera, which is reasonable.
2. The newly proposed dataset is much more diverse than previous efforts.
3. The proposed tactile-driven image stylization is interesting and inspiring.
4. Results on downstream tasks show the promising potential of the dataset.
weaknesses: I have two minor concerns.
1. The "approximately egocentric" video recording seems a little weird. The adopted camera (according to the photo in the supplementary materials) seems to require acceptable efforts to be made wearable. This makes the current recoding a little clumsy. I'm also having doubts about the stability of the videos recorded. Therefore, I'm curious about the motivation of the "approximately egocentric" video.
2. The visualization of tactile sensors is a little confusing. I'm having trouble understanding the meaning of the visualizations, e.g., the meaning of the dots and the colors.
correctness: I believe the claims in the submission are correct.
clarity: The paper is generally well-written.
relation to prior work: The authors clearly discussed its relation to prior works.
additional feedback: -
review_title: Generally good but having doubts on data collection and visualization

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a new dataset that helps to understand the material properties of objects in real-life. The authors use human data collectors to probe objects in the wild (both indoors and outdoors) and collect both video and information received on tactile sensors. The proposed dataset is an interesting one and has the potential to be utilized in many downstream tasks, however, its lack of standardization and potential bias in the data collection method might limit its use.
strengths: The visuo-tactile data has the following benefits:
- rigid object data can be collected by human
- probe objects in the wild (both indoors and outdoors)
- a wider range of objects
- a large quantity of data

weaknesses: - Tactile data collection lacks standardisation
The sequence of exploration procedures (EPs) that Lederman and Klatzky addressed in their experiments can have an effect on the tactile data collected from tactile sensors. The benefits of human-collected tactile data are outlined in the paper whereas the advantage of robotic data collecting remains unknown.  During the data collection, however, the **variable control** is unclear. The pressure or force was not indicated during the data collection process and was only mentioned in the paper as ''*do not perform actions with consistent force*''. This can bring large bias during collection, for example, the leather tactile data shown in the paper is collected by pressing the sofa while synthetic fabric by a cushion, both of them have foam cushion inside, without a constant force, the results can be varied and in reproducible. Lack standardisation could impede the ability of future researchers to continue building the datasets as well. A tactile sensor with a force sensor that indicates a force or force range could be valuable and help with standardisation here.

- The sensor might introduce a data collection bias
All data collected in this paper use the same sensor, it is not clear what the potential bias and limitations are in this sensor and the authors did not present a study on that. In addition, potential users with their own tactile sensors are very likely to suffer from the data bias because of the difference in sensing devices.

- Video recording
The human-centric view has its novelty. However, according to the image in the paper, these recordings may have **tracking registration errors**. In other words, neither the distance between the camera and the object nor the distance between the object and the sensor are constant. A focal point for the sensor in the video recording may provide a solution.
correctness: Please see the above section in Weakness.
clarity: The paper is well-presented with a clear comparison to some of the existing work. As mentioned previously, it would be nicer if the authors can elaborate on their tactile sensor and the potential limitations that this sensor might have.
relation to prior work: When compared with other visuo-tactile datasets, the author has simply categorised the collection methods as human or robot, the collection methodology is unclear. According to related work, from my understanding, the task of the human collection can refer to grasping by a robot (not clear), however, the validation part jumps to map with object manipulation. In addition, the robotic manipulation tasks are unclear in the related work. Besides the data collection method comparison, the author could also consider adding tactile sensor comparison as well. 

additional feedback: I do not have any additional feedback.
review_title: A human centric visuo-tactile data collection but lacks standardrization

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new dataset that contains both the visual and the tactile modalities for visuo-tactile learning. Different from the existing datasets that also contain tactile data, the new dataset is collected in natural environments with diverse objects. Experiments on three tasks have demonstrated the usefulness of the dataset.
strengths: - This paper proposes an interesting dataset that associates sight with touch. Compared to prior datasets that collect tactile data in controlled lab settings with limited objects or in simulation, this dataset collects real-world tactile data from natural environments. 

- Three interesting tasks have been performed to demonstrate the usefulness of the collected dataset: including 1) self-supervised vsual-tactile feature learning, 2) tactile-driven image stylization, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.

- Table 1 nicely summarizes the advantages of the proposed dataset. The new dataset is more human-centered, and it contains more diverse objects in diverse environments.
weaknesses: - It is motivated that one of the key advantages of this dataset is that it's human-centered, and "the work is a step toward human-collected multimodal data collection...., physically interacting with the world". While the environments where the data are collected from are indeed pretty diverse, the way to collect the data is still not how we humans touch objects in the world (with fingers). The data collection is done with a GelSight sensor. As shown in the qualitative illustrations, a human hand is grasping the sensor to press on different surfaces. It would to add some discussions on this. And why human-collected dataset is better?

- The markers (black dots) seem very noisy and not very well-calibrated. Some are large, and some are small. I wonder if there is any reason for this.
correctness: Yes, the dataset is constructed in a sound way. One question/concern is the quality of the tactile data (see weakness on the question on markers).
clarity: Yes, the paper is generally well written with nice illustrations.
relation to prior work: Yes, the paper clearly discusses its relation to prior work and compares with prior datasets in Table 1.
additional feedback: - Minor typo, L105, moveed -> moved
review_title: Valuable new dataset with three interesting applications on visuo-tactile learning

final_score: rating: 3: Clear rejection
review_content: summary and contributions: The paper introduces a database with videos of a webcam and gelsight camera output captured when a person probes objects in natural environments with a tactile sensor. It is more diverse than existing databases, and provides multi-modal information, but how these benefits touch and sight association tasks are unclear.

The main contribution is the dataset with labels of touching / not touching and different materials, which can be used for visuo-tactile learning problems.
strengths: The dataset introduces more diversity than prior databases. It also introduces a view of a person probing objects, which keeps more information than databases with only information related to the tactile sensor.
weaknesses: It is unclear how the dataset benefits from collecting more surfaces and keeping video of humans probing objects. The authors also claim that the dataset can help understand material properties, but this is not well proven in the paper.
correctness: The authors argue that many aspects of the visuo-tactile world are currently best learned by observing physical interactions performed by humans. This is demonstrated by adding more material to the dataset and adding videos of human probing objects, but the experiments in Chapter 5 show only minor improvement by involving the video in the learning tasks.

Another argument is that a dataset with more diversity can help with self-supervised learning and tasks that require an understanding of material property, but experiments in Chapter 5 lack comparison with more or less diversity during learning problems, and the understanding of material property is demonstrated by material classification, which has two major problems: the accuracy is only around 35%, and features of materials(roughness, hardness, etc.) are not extracted.
clarity: The paper is well written.
relation to prior work: It is clearly discussed how this work differs from the preivous contribution in Table 1.
additional feedback: Adding more objects and multi-modal information to the dataset is great, but it should be carefully considered how these help with learning tasks. Also, the dataset is more likely to be used in robot manipulation tasks, so it may be helpful to find out what information is essential to the tasks and then adjust the dataset.
review_title: A thorough database for touch and sight association tasks, but it is questionable how much benefit the database provides to the tasks.

Meta-review: metareview: Overall, all reviewers see the novelty in the dataset that provides an original new set of tactile data, accounting for a wide range of objects and scene, collected in the wild. The paper is overall well written, clear and easy to follow, and provides a good demonstration of the importance and usefulness of the dataset, going beyond datasets collected by robots in the context of material properties. It also includes initial steps towards benchmarking, although it is not the core contribution of the paper, which lies in the creation of the dataset itself. 

Two reviewers recommend acceptance, two stand marginally below acceptance, and one does recommend rejection. The authors have provided detailed responses to all reviewers’ comments and concerns and updated their manuscript. The main concerns of those reviewers sitting on the fence seem to be answered and limitations acknowledged however the element of bias in the data collection was still considered as a possible concern. Based on my reading and accounting for the authors changes, there approach itself is novel and original and hence provides already a relevant contribution to NeuroIPS; while the authors acknowledge limitations and future work avenues. The one reviewer recommending rejection didn’t engage in the discussion but from my reading of both the paper, the comments, and the authors responses, I believe the main points were addressed by the authors and misunderstandings clarified. Overall, I believe the papers offers a novel, non-existing dataset that can inspire interesting future works; all the dataset is shared and presented clearly for others to use. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept