Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper aimed to tackle the problem that Multisensory object-centric perception is limited by the small set of objects available. It presented OBJECTFOLDER, a dataset of 100 virtualized objects with implicit visual, auditory, and tactile representations. The usefulness of the proposed dataset was evaluated on an array of benchmark for both perception and control.

strengths_and_weaknesses: Strengths:
- For robotic applications, it is very expensive to obtain large-scale multisensory dataset, especially auditory and tactile data. The limitations have long been noticed. This paper proposed a novel method to obtain visual, auditory, and tactile sensory data of objects from virtual environments, which is well motivated.

- The paper introduced OBJECTFOLDER, a dataset of 00 virtualized objects that makes multisensory learning with vision, audio, and touch easily accessible to the research community.

- OBJECTFOLDER employs a uniform, object-centric, and implicit representation for each object’s visual textures, acoustic simulations, and tactile readings, making the dataset flexible to use and easy to share.

Weaknesses:
- Although the dataset contains100 object-centric visual, auditory, and tactile sensory data, the reviewer has doubts that whether such dataset can well represent auditory, and tactile attributes of the real objects. This should be discussed appropriately.

- The paper mentioned that the rendered images, impact sounds, and touch readings well match the ground-truth simulations. If only based on the results in Figure 4, this conclusion may be too arbitrary. It needs more experimental analysis to support this conclusion. 

- The paper claims the contribution that all objects in the proposed dataset are platform agnostic, which may not be a valid contribution. As a testbed, it should be the basic requirement.

- The paper adopted a variety of benchmark tasks to demonstrate the validity of the data set, which is unconvincing. The paper should verify whether the dataset is valid for robot perception and learning task in real-life scenarios. 

summary_of_recommendation: Although the method of obtaining visual, auditory, and tactile sensory data of objects from virtual environments is helpful for robotic multimodal learning, the presented experimental analysis did not effectively demonstrate the contribution of the paper. The paper needs to be further modified by adding additional experiments to improve the quality. Therefore, it is not ready for acceptance.

issues: - I have doubts that whether such dataset can well represent auditory, and tactile attributes of the real objects. Therefore, additional experimental analysis need to be presented to show that the multisensory data can well represent object attributes.

- The paper mentioned that the rendered images, impact sounds, and touch readings well match the ground-truth simulations. It needs more experimental analysis to support this conclusion. 

- According to the focus of the paper, it needs to redesign the experiments to prove the validity of the dataset. In particular, some experiments need to be added to verify effectiveness of the dataset for robot perception and learning task in real scenarios.

- The dataset is evaluated on four different benchmark tasks. This resulted in that experimental details are ignored and the experimental analysis is not sufficient. I think that focusing on one benchmark task can improve the clarity and completeness of the paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors present ObjectFolder, a dataset containing 100 objects with implicit visual, audio and tactile representations. A novel method is proposed for representing the audio signals implicitly. Once this dataset is built, the implicit functions are evaluated in a series of passive benchmarks measuring 3D reconstruction quality, cross-modal instance recognition/retrieval and grasping stability prediction in simulation. 
strengths_and_weaknesses: First of all, I must say, this paper was a joy to read and is very original. The approach is very novel and could potentially be useful for the robotics community which has traditionally focused on explicit data representations with mixed results.

VisionNet and TouchNet seem to have limited contributions since they are mainly inspired from NERF. The representation for AudioNet on the other hand seems to be a novel contribution.

The biggest concern for the paper is that there is no comparison to explicit methods. For instance, a direct comparison to the network architecture in Calandra et al. [CoRL 2017] would be insightful for the grasp stability experiment. Without this comparison, as a reader I was still wondering why a implicit representation is functionally necessary over an explicit one.

Also it is challenging to understand if the implicit representation transfers well to real world tasks since the results were only passively tested (i.e. without contact-rich manipulation) or in simulation. 

Table 3 is very insightful. It shows that vision and audio are good modalities to capture global information (like the object instance or its category) compared to tactile (which mainly has local geometric information).
summary_of_recommendation: I support to accept this paper in its current form. Despite my reservations on the potential impact and widespread adoption of the dataset in the community, there are definitely novel contributions in the idea and architecture.
issues: See above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a dataset of 100 virtual objects with visual, auditory, and tactile sensory input for learning multi-modal object representations. The dataset is constructed from scanned real-world objects available from ModelHaven, YCB, and Google Scanned Objects. The audio is synthesized through rigid-body sound simulation and touch is simulated with TACTO for emulating a DIGIT sensor. To investigate the usefulness of multimodal representations, the paper presents a model for learning neural representations with three subnetworks – VisionNet, AudioNet, and TouchNet. Evaluations on downstream tasks such as instance recognition, cross-sensory retrieval, and grasping, show that using multimodal representations result in significant performance gains over simple unimodal or bimodal baselines.
strengths_and_weaknesses: Strengths: 
- This paper makes an important resource contribution to robotics. Most robot learning methods heavily rely on vision for perception. There has been overwhelming evidence that perception for manipulation is inherently multimodal and to learn robust and dexterous skills, robotic systems need to go beyond just visual perception. 
- Benchmarking and reproducibility are crucial for evaluating progress in various robotic domains. Having a standardized set of objects for multimodal perception, particularly in a simulation, is very beneficial for benchmarking future works in detection, grasping, and dexterous manipulation. To the best of my knowledge, this is the first benchmark that provides all three modalities – vision, audio, and tactile, in a simulated setup.
- The evaluations on downstream tasks are insightful. In all cases from detection to grasping, the multimodal representations outperform unimodal and bimodal baselines. 

Weaknesses: 
- One concern, although relatively minor, is the quality of the audio and tactile simulation and their fidelity to real-world sensors. While the visual rendering of scanned objects is near photorealistic, the audio simulation uses rigid-body simulation to emulate audio sensors. For this simulation, each object is modelled with a single material since individual parts of the scanned objects are unknown. This could be an issue for some objects, for example, striking a hammer on the plastic handle might produce a different sound than striking it on the metallic head. This might also explain why the Audio-only baseline in Table 2 performs so well. Similarly, the simulated tactile sensor uses the geometric structure of scanned objects which could have some imperfections and sharp edges from 3D reconstruction. Ideally, both these issues can be addressed by evaluating the simulated sound and tactile inputs against real-world sensor data. This can be done by picking a few objects from the dataset, collecting data with real-world objects, and comparing it against the simulated audio and tactile sensors.
- Another concern, noticeable in the supplementary video, is that the dataset contains a lot of cups or cup-like objects of cylindrical shapes. Was this a specific design choice? Or a symptom of what was available/feasible for robot grasping? Since the audio and tactile inputs are automatically simulated, is there anything preventing the dataset from containing more than 100 objects? Perhaps all of Google Scanned Objects and ModelHaven objects? Benchmarks need to be careful not to induce biases in the dataset that might result in models overfitting or exploiting biased data without learning how to solve the task. 

summary_of_recommendation: This paper makes an important resource contribution with a multimodal dataset of virtual objects. Experimental evaluations on various downstream robotics tasks indicate that multimodal representations are crucial for going the last mile in detection, grasping, and related tasks. 

**Post Rebuttal**  
I am happy with the authors' response and followup experiments, so I am keeping my original score. I'd encourage the authors to add more diverse objects to the dataset, if possible.
issues: Minor comments:
- From the supplementary video, the center-of-mass (COM) of Google Scanned Objects seem to be incorrect, leading to wobbly objects that never fall down. This is a known issue with Google Scanned Objects where the COM is at the bottom of the object rather than it’s geometric center or true center. Accordingly, you probably need to re-run the grasping experiments with the corrected COMs.
- How many training images were used to train the NeRF? How long does it take to train a NeRF per object?
review_title: null

Meta-review: This paper proposes a new dataset of 100 synthetic objects with visual, auditory, and tactile sensory features for learning multi-modal object representations. The dataset combines objects from the YCB dataset, the ModelHaven, and the Google Scanned Objects. The benefits of the proposed dataset as a testbed for multisensory perception and control are demonstrated by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.
The reviewers agree that the proposed dataset can be useful for pushing ahead the state of the art in robotic grasping and manipulation. A key issue however is the realism of the dataset, which is not clear from this paper. The authors should demonstrate the benefits of using this dataset on real robotic manipulation tasks. 
Decision: Accept (Poster)