ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Authors: Ruohan Gao
Yen-Yu Chang
Shivani Mall
Li Fei-Fei
Jiajun Wu
Venue: CoRL2021 Poster (2021)
Keywords: object dataset
multisensory learning
implicit representations

Abstract: Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years. However, the progress in these directions is limited by the small set of objects available---synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and financial cost.  We present ObjectFolder, a dataset of 100 virtualized objects that addresses both challenges with two key innovations. First, ObjectFolder encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognition tasks, beyond existing datasets that focus purely on object geometry. Second, ObjectFolder employs a uniform, object-centric, and implicit representation for each object’s visual textures, acoustic simulations, and tactile readings, making the dataset flexible to use and easy to share. We demonstrate the usefulness of our dataset as a testbed for multisensory perception and control by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.

OBJECTFOLDER: A Dataset of Objects with Implicit
Visual, Auditory, and Tactile Representations
Ruohan Gao
Yen-Yu Chang"
Shivani Mall
Li Fei-Fei
Jiajun Wu
Stanford University
Abstract:
Multisensory object-centric perception, reasoning, and interaction have been key research topic in recent years. However, the progress in these directions is limited by the small set of objects caallable-syntheen objects are not realis- tic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable acquire due to in- ternational shipping, inventory, and financial cost. We present OBJECTFOLDER a dataset of 100 virtualized objects that addresses both challenges with two key innovations. First, OBJECTFOLDEER encodes the visual, auditory, and tactile sen- sory data for all objects, enabling a number of multisensory object recognition tasks, beyond existing datasets that focus purely object geometry. Second, OBJECTFOLDER employs a uniform, object-centric, and implicit representation for each object's visual textures, acoustic simulations, and tactile readings, mak- ing the dataset flexible to use and easy to share. We demonstrate the usefulness of our dataset it on a variety benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.
testbed for multisensory perception and control by evaluating
Keywords: object dataset, multisensory learning, implicit representations
ObjectFolder
Multisensory Perception and Control
Mug
Vision
Extrinsic
Audio
Instance recognition
Cross-sensory retrieval
parameters
Touch
Object File
3D reconstruction
Robotic grasping
Figure OBJECTFOLDES contains 100 Object Files the form of implicit neural representa- tions. Querying each Object File with the corresponding extrinsic parameters object-centric visual, auditory, and tactile sensory data. The OBJECTFOLDER dataset is potential testbed for many machine learning and robotics tasks that require multisensory perception.
obtain realistic
Introduction
We perceive the world not as single giant entity but often through a series inanimate objects, which exist as bounded wholes and move on connected paths. We interact with these objects through an array of different sensory gystems-vision audition, touch, smell, taste, and proprioception. These multisensory inputs shape our daily experiences: we observe the surroundings to avoid obstacles, hear the doorbell ring realize the guests have arrived, and touch the fabric to sense its degree of comfort. Cognitive science studies [1, 2] show that both object representation and multisensory perception play a crucial role in human cognitive development.
Object-centric learning has shown great potential lately in visual reasoning [3], representing physi- cal scenes [4], modeling multi-agent interactions [5], and improving generalization to unseen object
indicates equal contribution.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

compositions [6]. These prior methods, however, often do not model the full spectrum of physical object properties and sensory modes, including their 3D shape. texture, material, sound, and feel. While there has been significant progress by "looking" recogniiing objects based on glimpses of their visual appearance or 3D ohjects in the world are often modeled as silent and untouch- able entities.
We identify that this due the lack of rich object datasets. Existing object datasets have con-
tributed to most significant progress in image recognition [7. 8] and shape modeling [9, 10), but other sensory information is minimally considered. Datasets for robotic manipulation research [11, 12] consist of a selected number of real-world household objects, but these objects are expensive and often unstable to acquire due to international shipping and inventory. Moreover, is nontrivial to virtualize these objects and their multiple sensory modes, limiting their usability for developing embodied AI agents [13, 14, 15, 16, 17, 18, 19, 20, 21].
Our goal is therefore to create a dataset of 3D objects that are 1) easily accessible to the community as standard benchmark, 2) high-quality in terms of visual textures, and 3) augmented with realistic auditory and tactile sensory data. Towards this end, we introduce OBJECTFOLDER-4 multisensory dataset of implicitly represented Object Files. The concept of an object file can be traced back to Kahneman et al. [22], where it is defined as the sensory information that has been received about the object at particular location. In similar spirit, we model and represent each object (or its intrinsics) using an implicit neural representation, which through querying with extrinsic parameters we can obtain images of the object from different viewpoints, impact sounds of the object each position, and tactile sensing of the object at every surface location. These different sensory data of vision, audio, and touch can be regarded as contemporary multisensory object files.
Specifically, we collect 100 high quality 3D objects from online repositories. For each object, we model its visual appearance at random object pose, lighting condition, and camera viewpoint; we perform modal analysis [23] based on its shape, size, and material type to calculate its characteris- tic vibration modes for audio simulation; and we simulate its touch readings at all surface locations using DIGIT [24. 25 vision-basee tactile sensor. Based on recent success on neural implicit repre- sentations [26, 27]. we design a deep neural network that consists of three sub-networks- Sob--nttorks-vsiionNeee AudioNet, and TouchNet, which encode the visual, auditory. and tactile sensory data for each object, respectively. See Fig. 3. Together, they constitute an Object File that contains the complete multi- sensory profile for each object instance. Furthermore, we demonstrate the usefulness of our dataset on four benchmark tasks leveraging multisensory data, including instance recognition, cross-modal retrieval, 3D reconstruction, and robotic grasping.
First, we introduce OBJECTFOLDER, dataset that makes multisensory learning with vision, audio, and touch easily accessible to the research community; Second, all objects in our dataset are compatible to different robotic virtual environments and will be made publicly available as standard testbed for robotic multisensory learning; Third, we evaluate on a suite of benchmark tasks that require multisensory data to facilitate future work in this direction.
Our main contributions are threefold:
2 Related Work
Object-Centric Datasets Many image datasets exist for object recognition, such as ImageNet [7], MS COCO [28], ObjectNet [8], and OpenImages [29]. These datasets consist of 2D images, while ORJECTFOLLDER is dataset of 3D objects. ModelNet [9] and ShapeNet [10] are two large-scale datasets of 3D models, but the collected 3D CAD models either contain no or low-quality visual textures, making them unsuitable for real-world applications. For research on object manipulation, datasets are constructed using real-world household objects such as YCB [11] and BigBIRD [12]. Different from them, our goal is to construct dataset of 3D objects, where realistic visual, auditory, and tactile sensory data can be easily obtained for each object in virtual environments.
Goordinate-based multi-layer perceptrons (MLP) have recently been adopted as continuous, memory-efficient implicit representations for a variety of visual sig- nals such as 3D shape [30, 31], scenes [32, 33], and object appearance [26, 27] with the help of classic volume rendering techniques [34]. We also use MLP as a compact neural representation to represent each 3D object. Differently, apart from visual appearance, our mutisensory implicit neural representation also encodes object-centric auditory and tactile data of the object.
Implicit Neural Representations
Multisensory Perception for Robotics Recent work shows growing interests in using audio or touch in conjunction with vision for robotic tasks. Audio is used to recognize object instances [35]
2

---

Figure 2: Example objects in OBJECTFOLDER. See Supp. for the visualization of the entire dataset.
or terrain types [36]; estimate the flow and amount of granular materials [37] or liquid height [38] in robotic scooping and pouring tasks; study its synergies with the motion of a robot [39]; and bridge the simulation-to-rcality gap for tasks that involve stochastic dynamics [40]. Tactile sensing more local compared to vision and audio, and it is shown improve sample efficiency on peg insertion task [41]; benefit robotic grasping [42, 43, 44]; augment vision to enhance 3D shape recon- struction [45]. Different from the above work that focuses on particular task that leverages either auditory or tactile data, we construct an object-centric dataset of all three modalities, democratizing multisensory learning research using vision, audio, and touch.
3 OBJECTFOLDER
We introduce OBJECTFOLDER, a dataset of 100 implicitly represented 3D) objects with vision, au- dio, and touch. As shown in Fig. 3, each object is represented by Object File, a compact neural network that contains all the visual, acoustic, and tactile profiles for the object. Querying it with ex- trinsic parameters we can obtain visual appearance of the object from different viewpoints, impact sound of the object at each position, and tactile reading of the object at every surface location. In the following, we first describe the source of our objects and the annotations (Sec. 3.1). Then we present the simulation pipeline and how we encode the sensory data using implicit neural representations for vision (Sec. 3.2), audio (Sec. 3.3), and touch (Sec. 3.4), respectively.
3.1 Objects
We collect 100 high quality 3D objects from online repositories including: 20 objects from 3D Model Haven?, 28 objects from the YCB dataset', and 52 objects from Google Scanned Objects*. We select objects that are of realistic visual textures, approximately homogeneous material property, and with a material type that mappable to one of the following categories: ceramic, glass, wood, plastic, iron, polycarbonate, and steel. We annotate each object with the material type, which will be used in audio simulation in Sec. 3.3. The dataset contains common household objects of diverse categories such as bowl, mug, cabinet, television, shelf, fork, and spoon. See Supp. for details.
3.2 Vision
Simulation: We use Blender's Cycles path tracer [46] to render images. For each object, we first normalize it into unit cube and use point light source at a random location on unit sphere. We then render images of the object on white background from camera viewpoints randomly sampled on a full sphere. See Fig. 2 for some example views of objects in our dataset.
After rendering images from different camera viewpoints under varied lighting con- ditions, we use VisionNet to encode the visual appearance for each object. Following prior work on object-centric neural radiance fields [27], represent each object as 7D object-centric neu- ral scattering function whose input is 3D location x (æ, y, 2) in the object coordinate frame and the lighting condition at that location (W,,Wo), where ai (Ø., f,), wo (Φ.. (0..0.) de- note the incoming and outgoing light directions, respectively. The output is the volume density and fraction of the incoming light that is scattered in the outgoing direction P (pr: Pg: Pb)- VisionNet approximates this continuous 7D object-centric representation with an MIP network (a, Pr. Pg: (b) that maps each input 7D coordinate to its correspond-
VisionNet:
F, (ar, y, 2, 02, Φs, fo, 6o)
"httpe://3dmodeelaave
20ScamnedX200bjjects

---

Extrinsics
Object File (Intrinsics)
VisionNet
3D coordinates (x,
densily
Volume rendering
Light directions (Ø, O2 00. 0.)
color (prr P Po)
AudioNet
kg
19111
X
k2
Spectrogram location t)
ISTFT X
Modes addition
ISTFT V
Tactile mapping
TouchNet
Tactile map
location (w, ht)
Figure 3: Each Object File implicit neural representation contains three Doo-networks--isiine AudioNet, and TouchNet, which through querying with the corresponding extrinsic parameters we obtain the visual appearance of the object from different views, impact sounds of the object each position, and tactile sensing of the object at every surface location, respectively.
ing volume density and fraction coefficient for each RGB channel. The amount of light scattered a point x can be obtained as follows:
Lg Lx(x, = . L(x, wo)
where S is unit sphere, L(x,w; denotes the amount of light scattered point x along direction wis and fp evaluates the fraction light incoming from direction wi at the point that scatters out in direction wo. Then through the object. The expected color Crr) of camera ray r(l) Xn + lwa can be obtained as follows:
classic volume rendering [34] render the color of any ray passing
c(r)=-(" ff (()) wo)dt where T(t) = exp( - f. (r((s))
(2)
T()) denotes the accumulated transmittance along the ray from ln to 1; L and l are the near and far integration bounds; and (r((t) denotes the volume density location r(t). Similar to [26], we also use stratified sampling, positional encoding, and hierarchical volume sampling to increase rendering quality and efficiency. See [26, 27] for details.
3.3 Audio
Simulation: The goal of audio simulation is to realistically model the impact sound at each posi- tion of the object based its shape, size, material type, external force, and the contact location. Following the standard approach in engincering and acoustics, we use linear modal analysis for physics-based rigid-body sound synthesis [23, 47].
Firstly, we convert the surface mesh an object into a volumetric hexahedron mesh, which repre- sents the original shape with N voxels. The key to modal analysis is to solve the following linear deformation equation for a 3D linear elastic dynamics model:
Mü + Cú Ku=
where E R3N denotes the displacement of elemental nodes in 3D: M.K E IaNNaN denote mass and stiffness matrices of the system; f E R3N represents the external force applied to the object that stimulates the vibration. Based on the material type of the object, we obtain density value p. Young's Modulus F, Poisson's ratio r, and Rayleigh damping parameters and B. The scale of the object and material parameters p, E.r
aM + 3K stands for Rayleigh damping; and

---

are used to build the mass and stiffness matrices M and K. See Supp. for how we map each material type to these parameters.
The above equation can be decoupled into the following form through generalized eigenvalue de- composition KU AMU:
a (aI + 3A)a + Aq= = U
(4)
where u = Uq and A is diagonal matrix. The solution to the above equation a bank of damped sinusoidal waves with each wave representing mode signal:
9: a,e a.e (2w,t + 8z),
(5)
where wa is the frequency of the mode, e is the damping coefficient, o; is the excited amplitude, and 0: is the initial phase. Assuming the object is stationary originally, we get the following solution:
4: = (a + (Ar); wi -
0. =
2
X represents the in mode eigenvalue. We simulate the vibration modes excited by unit forces f1, f2. f3 for all vertexes and use the obtained modes signals for training AudioNet as detailed next.
AudioNet: As shown in Fig. 3, for each force direction, we train a separate branch to encode the corresponding modes signals at all vertexes. We obtain the audio spectrogram" for each modes signal and train a MLP to predict the complex spectrogram of dimension F x T x 2, where F and T are the frequeney and time dimensions. Partieularly, each branch takes the spatial coordinates (I, y: 2) and the spectrogram location (f. t) as input, and predict the real and imaginary part of the complex number for every location in the audio spectrogram. The modes signal can then be recovered from the spectrogram using inverse short-time Fourier transform (ISTFT). We also use positional encoding as in VisionNet.
During test time, we directly use vibration modes for sound synthesis following [47]. Any external force at vertex can be decomposed into linear combination of unit forces along the three orthogo- nal directions: = k2 f + kz + kigfs. The amplitudes of modes a excited by f can be decomposed into linear combination of the amplitudes excited by the unit force: where a1, a2, ag denote the amplitudes of modes excited by unit forces f1, f2, fa for a given vertex. Because modal analysis is performed on a volumetric hexahedron mesh, for a vertex in the original polygon mesh, we find its closest four vertexes in the hexahedron and average their modes signals.
kja1 + k2a2 + kyas,
Prior work [23, 48] have demonstrated that such physics-based rigid-body sound synthesis pipeline renders sound that transfers well to real scenes and the objects' shape, material, and other auditory attributes can be successfully inferred from the simulated sound. To further demonstrate the realism of our audio simulation, we also perform a user study on acoustic fidelity that asks participants to distinguish between the real and simulated audios. Results show that our simulated audio is preferred by 42% of the total responses, comparing closely to real audio recordings. See Supp. for details.
3.4 Touch
Simulation: We leverage a state-of-the-art touch simulator TACTO [25] for touch simulation. TACTO is vision-based touch simulator and uses ray-tracing to simulate high-quality tactile sig- nals. It can simulate realistic rendering for both contact surfaces and shadows with different contact geometry. See [25] for details and a comparison of the simulated and real tactile readings. Particu- larly, we use DIGIT [24] touch sensor and its corresponding camera, lights, and gel mesh configura- tions in simulation. We choose DIGIT for its compactness, high-resolution of its representation. and potential applications for robotic in-hand manipulation. Other touch sensors such as Gelsight [49], BioTac [50], or Ominitact [51] can also be potentially used, and the design of the touch sensor to use is orthogonal to our work. To obtain touch simulations, we place each object at rest and move DIGIT to touch each object vertex in vertex normal direction. We constrain the force to range within small threshold such that the rendered tactile images do not vary significantly with different forces. We use the RGB tactile image that contains the local contact geometry as our touch representation.
TouchNet: For each object, we obtain RGB tactile image of dimension W x H x 3 that captures the local geometry information for every vertex on the surface of the polygon mesh. TouchNet takes
'ee find predicting spectrogram works better than predicting modes signal directly. suspect spectrogram more structured by disentangling time and frequency dimensions, making the network easier to optimize. For audio simulation, we vertex spatial coordinates the hexahedron mesh for modal analysis.
5

---

Object File
Ground-Truth
Figure 4: Comparison of the visual, auditory, and tactile data generated from Object Files and the corresponding ground truth simulations. Our implicit neural representations accurately encode the sensory data for the objects. See Supp. for more examples and quantitative comparison.
the spatial coordinate (x, y, 2) of the vertex and the spatial location (w, h) in the tactile image as input, and predicts the per-pixel value for the three channels of the tactile RGB image. The other settings are similar to VisionNet and AudioNet.
Fig. 4 shows some examples of the visual, auditory, and tac- tile sensory data obtained from Object Files. The rendered images, impact sounds, and touch readings well match the ground-truth simulations. See Supp. for a quantitative com- parison the accuracy of the implicit representations. More- over, using implicit neural representations is much more stor- age efficient compared the original data format. Table shows a comparison between the storage size with and with- using implicit representations. Object File the form of an implicit neural network is a much more compact representation of the multisensory data for each object. Note that VisionNet models an object-centric neural scattering function [27] and thus can generalize to any camera viewpoint under arbitrary lighting conditions, while rendering and saving images for all scenarios takes infinite space theoretically.
Implicit
Original
Vision Audio Touch
7.2 MB
X
6.3 MB
12.6 GB
2.1 MB
2.9 GB
Table 1: Storage space comparison.
Experiments
OBJECTFOLDER is a potential testbed for many perception and control tasks. Next, we evaluate on four benchmark tasks including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping to demonstrate the usefulness of the dataset and the value of multisensory perception. We show the key results below, and see Supp. for the detailed experiments setup.
4.1 Multisensory Instance Recognition
Identifying the object instance that is interacted with is fundamental for many robotic applications. In this task, we want to identify an object based on either its visual appearance, impact sound. or local contact geometry. We are interested in finding out the amount of information that is useful recognize the object instance from each modality. For the vision modality, we train a ResNet-18 [52] network that takes an RGB image the object as input and predicts the instance label of the object. The settings are the same for audio and touch except that the input is either a magnitude spectrogram of the impact sound the object generates or a local tactile RGB image from a random location on the object surface.
Methods
Ace. (%)
Chance
1.00
Vision (V)
94.8
Audio (A)
98.3
Touch (T)
72.4
V A
99.5
V + T
97.2
A T V+A+T 99.8
99.0
Table 2: Results on multisen- sory instance recognition.
Table shows the results. The accuracy for a random classifier 1%. The vision classifier achieves high accuracy, because image of the object normally captures informative appearance cues that distinguish the object instance. Under some camera viewpoints, the vision classifier can mistakenly classify the object. Surprisingly, the audio classifier achieves the best results. This confirms that our audio simulation pipeline well models the shape, size, and material property of the object instance, so that single impact sound can be used to recognize the object. Touch is good at capturing local geometry of an object, but often does not contain sufficient discriminative cues to recognize the object especially from single touch. Therefore, the touch classifier performs worse compared vision and audio. Classifiers with multisensory input are more robust, and combining all three modalities leads to the best result.

---

4.2 Cross-Sensory Retrieval
Cross-sensory retrieval plays a crucial role in machine perception to understand the relationships between different modalities. Fig. 5 shows our framework for learning cross- cross-sensory embeddings for two modalities. Taking cross cross-sensory learning with vision and audio as an example, we design a two-stream network that takes an RGB image and an audio spectrogram of the same object as input for each stream. The two modalities have disjoint pathways in the early layers to capture the -speeif features. A final modality-agnostic network is used to map the shared embedding to the label space for training with cross-entropy loss. We map the features of both modalities from the last hidden state to a cross-sensory embedding space through triplet loss. We either sample an image audio from another object as the negative example, and pull the embeddings of the same object to be closer in the joint embedding space and push the embeddings from different objects apart. The full model is trained jointly with both the triplet loss and the cross-entropy loss.
Input
Retrieved
Chance CCA [53]
Ours
Modality A
Encoder A
Modality- Saailly--gnnstiiccc Net
Audio
0.05
0.57
0.90
Crose aenoory smbedding space
Vision
Classification
Touch
0.05
0.24
0.50
Negative
Vision
0.05
0.59
0.92
Push
Audio
Touch
0.05
0.31
0.55
Vision
0.05
0.29
0.48
Modality B - Encoder B
Touch
Audio
0.05
0.33
0.64
Figure 5: Learning cross-sensory embeddings.
Table 3: Cross-sensory retrieval results.
Table 3 shows the results for cross-sensory retrieval on the test set, where we retrieve the sensory data of a different modality by querying from the shared embedding space. We use cosine similarity measure the distances between embeddings and retrieve the sample with the smallest distance. We report mean Average Precision (mAP) which jointly considers the ranking information and the precision, and compare with random retrieval and a state-of-the-art cross-sensory retrieval method based on canonical correlation analysis (CCA) [53]. Our models perform more reliable retrievals, and the results show that the learned cross gross-sensory embeddings are effective and contain fine- fine-grained information about objects to correctly retrieve sensory data from the matched object. Similar to instance recognition, we observe vision and audio provide more reliable global information about the object compared to touch.
4.3 Audio Visual 3D Reconstruction
Robots often need to build a mental model of the 3D shape of objects based on just a single glimpse or the sounds they make during interaction. In this section, we investigate the potential of 3D recon- struction using either a single image, an impact sound, or their combination. We leverage Occupancy Network [30], which implicitly represents the 3D surface as the continuous decision boundary of deep neural network classifier, as a testbed for this task. Fig. 6 illustrates our IMAGE+ AUDIO2MEDD framework for audio-visual 3D shape reconstruction. The image and audio embeddings are fused through a fusion layer into an audio-visual feature, then combined with coordinate cvordinate-conditioneed conditioned fea- predict occupancy probabilities p. We also
ture maps through Conditional Fatch-Norrmalizttin which represents existing single image 3D recon-
test two of its simplified variants IMAGE2M. struction approaches that predict the 3D shape from a single view of the object, and AUDIO2MESH, which performs 3D reconstruction purely from audio.
Methods
IoU 1
Chamfer-Ly + Normal Consistency 1
AVERAGE
0.0675
0.1067
0.6312
IMAGE2MESH [30]
0.8809
0.0046
0,9522
AUDIO2MESH
0.8729
0.0048
0.9504
IMAGE-AUDIOMESSH
0.8906
0.0043
0.9535
Table 3D shape reconstruction results. 4. lower better, t higher better.
We evaluate on a held out test set with standard metrics: IoU, Chamfer-L,, distance, and Normal Consistency with respect to the ground-truth mesh. Table 4 shows the results. Compared to the
7

---

simple AVERAGE baseline that averages the results obtained using the ground- truth mesh of each of the 100 objects as the prediction, we can see that 3D reconstruction from single image or an impact sound performs much better. Augmenting traditional single image 3d reconstruction with audio achieves the best performance by leveraging the additional acoustic spatial cues. Furthermore, as shown in Fig. 7, we apply our IMAGE2MESH model trained on objects in OBJECTFOL.DE to real-world images. Our model generalizes reasonably well to real image, demonstrating the realism of the objects in our dataset. The last column shows typical failure case, where the teapot image is very visually different from objects in our dataset.
Image
Encoder
Fusion layer
Input
Occupancy
Network
Audio
Encoder
Reconstruction
(x,
Figure 6:
AUDGO+AUDIO2M
Figure 7: 3D shape reconstruction using real-world im-
framework for 3D reconstruction.
ages. Last column shows a typical failure case.
4.4 Robotic Grasping with Vision and Touch
Next show how combine vision and touch for prediction of grasp stability. The goal is to use a robotic arm to successfully grasp and hold an object between the robot's 's left and right fingers. For touch data, infer the RGB tactile image from the corresponding Object File based on the grasp contact positions of the left and right grippers on the object; For vision data, we obtain images from an externally mounted camera's viewpoint at the grasping moment. We label the data as either "Success" or "Failure" based on whether the object can be held between the robot's fingers after being lifted. Fig. 8 shows example of both successful grasp and failure case.
We collect data for 10 objects from OBJECTFOLDER, and then train binary classifiers under different sample sizes predict the grasp outcome based on either the vision signal, the touch signal, their combination. This experiment is similar to that of [44, 25] except that we obtain touch readings directly by querying implicit representation networks. Fig. shows the results. We observe that takes significantly less amount of data to reach high accuracy when learning from touch compared to vision. Leveraging both visual and tactile data achieves the best accuracy. Additionally, train a policy for touch-based robotic grasping using TRPO [54]. We achieve a success rate of 75.5% during testing, while that of random grasping policy 53.0%. To further demonstrate the potential of using our dataset on robot manipulation tasks, we also perform an experiment on an object manipulation task (reach) using Meta- Meta-World [55] with three of our objects (cup. bowl, dice). We achieve 100% success rate for each object. See Supp. for details.
Successful grasp
Failed grasp
Vaiio
Touch Vision Touch
Figure 8: Examples of successful grasp and failed grasp. L: left finger, R: right finger.
Number sampless
Figure Grasp stability prediction.
Conclusion
We presented OBJECTFOLDER, dataset of objects with implicit visual, auditory, and tactile rep- resentations. It will facilitate research with vision, audio, and touch, and enable multisensory sim- ulation of objects robotic virtual environments. Through evaluation on an array of benchmark tasks, we show the usefulness of our dataset for both perception and control. As future work, we plan to explore richer object states with more accurate and grained physics, additional sensory modalities, and textual descriptions.
8

---

Acknowledgemmnts:: We thank Michelle Guo, Xutong Jin, Shaoxiong Wang, Huazhe Xu, and Samuel Clarke for helpful discussions on experiments setup and Michael Lingelbach and Jingwei Ji for suggestions on paper drafts. The work is in part supported by Toyota Research Institute (TRI), Samsung Global Research Outreach (GRO) program, ARMY MURI grant W911NF-15-11 0479, NSF CCRI #21 20095, Amazon Research Award (ARA), Autodesk, Qualcomm, and Stanford Institute for ClmannCCenters AI (HAI).
References
[T] E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 2007.
[2] L. Smith and M. Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 2005.
[3] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In ICLR, 2020.
[4] D. M. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. Fei-Fei, J. Wu, B. Tenenbaum, et al. Learning physical graph representations from visual scenes. In NeurIPS, 2020.
[5] C. Sun, P. Karlsson, J. Wu, J. B. Tenenbaum, and K. Murphy. Stochastic prediction of multi- agent interactions from partial observations. ICLR, 2019.
[6] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. In NeurIPS, 2020.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In CVPR, 2009.
[8] A. Barbu, Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurlPS, 2019.
[9] Z. Wu, S. Song, A. Khosla. F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. CVPR, 2015.
[10] A. X. Chang, T. Funkhouser, I.. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
[11] B. Calli, A. Singh. A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In ICRA, 2015.
[12] A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbeel. Bigbird:(big) berkeley instance recognition dataset. In ICRA, 2014.
[13] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, Malik, D. Parikh, and D. Batra. Habitat: Platform for Embodied AI Research. In ICCV, 2019.
[14] Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang. Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In CVPR, 2020.
[15] B. Shen, F. Xia, C. Li, R. Martín-Martin L. Fan, G. Wang, S. Buch, C. D' Arpino, S. Sri- vastava, L. P. Tchapmi. et al. igibson, simulation environment for interactive tasks in large realisticscenes. arXiv preprint arXiv:2012.02924 2020.
[16] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, F. Kolve, R. Mottaghi, Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, I.. Weihs, M. Yatskar, and A. Farhadi. RoboTHOR: An Open Simulation-to-Reet Embodied AI Platform. In CVPR, 2020.

---

[17] X. Puig, K. Ra, M. Boben, J. Li, T. Wang. S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs, In CVPR, 2018.
[18] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, Kubilius, A. Bhandwal- dar, N. Haber, M. Sano, et al. Threedworid: A platform for interactive multi-modal physical simulation. arXiv preprint urXiv:2007.04954 2020.
[19] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
[20] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. RA-L, 2020.
[21] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and
K. Grauman. Soundspaces: Audio-visual navigaton in 3d environments. In ECCV, 2020.
[22] D. Kahneman, A. Treisman, and B. J. Gibbs. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 1992.
[23] Z. Ren, H. Yeh, and M. C. Lin. Example-guided physically based modal sound synthesis. TOG, 2013.
[24] M. Lambeta, P.-W. Chou, S. Tian, B. Yang. B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. RA-I, 2020.
[25] S. Wang, M. Lambeta, P.-W. Chou, and R. Calandra. Tacto: A fast, flexible and open-source simulator for high-resolution vision-based tactile sensors. arXiv preprint arXiv:2012.0844 084456, 2020.
[26] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] M. Guo, A. Fathi, Wu, and T. Funkhouser. Object-centric neural scene rendering. arXiv preprint 2rXiv:2012.08500 2020.
[28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[29] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, al. The open images dataset v4. IJCV, 2020.
[30] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[31] J. Park, P. Florence. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning contin- uous signed distance functions for shape representation. In CVPR, 2019.
[32] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous di-sstuccure-wwart aware neural scene representations. In NeurlPS, 2019.
[33] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. arXiv preprint arXiv:2003.046/1 046688 2020.
[34] J. T. Kajiya and B. P. Von Herzen. Ray tracing volume densities. SIGGRAPH, 1984.
[35] J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. In ICRA, 2009.
[36] J. Christie and N. Kottege. Acoustics based terrain classification for legged robots. In ICRA, 2016.
[37] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. In CoRI., 2018.
10

---

[38] H. Liang, S. Li. X. Ma, N. Hendrich, T. Gerkmann, F. Sun, and J. Zhang. Making sense of audio vibration for liquid height estimation in robotic pouring. In IROS, 2019.
[39] D. Gandhi, A. Gupta, and L. Pinto. Swoosh! rattle! thump!-actions that sound. In RSS, 2020,
[40] C. Matl, Y. Narang, D. Fox, R. Bajesy, and F. Ramos. Stressd: Sim-to-real from sound for stochastic dynamics. In CoRL, 2020.
[41] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, S. Savarese, L.. Fei-Fei, A. Garg, and J. Bohg. Making sense vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA, 2019.
[42] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, Malik, E. H. Adelson, and Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. RA-L, 2018.
[43] L. Pinto, D. Gandhi, Y. Han, Y.-.. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
[44] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine. The feeling of success: Does touch sensing help predict grasp outcomes? In CoRL, 2017.
[45] E. J. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. In NeurIPS, 2020.
[46] Blender a 3d modelling and rendering package. attp://wwn http:://wwwbbbenee. blender org.
[47] X. Jin, S. Li, T. Qu, D. Manocha, and G. Wang. Deep-modal: real-time impact sound synthesis for arbitrary shapes. ACMMM, 2020.
[48] Z. Zhang, Wu, Q. Li, Z. Huang, J. Traer, H. McDermott, J. B. Tenenbaum, and W. T. Freeman. Generative modeling of audible shapes for object perception. In ICCV, 2017.
[49] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 2017.
[50] N. Wettels and G. E. Loeb. Haptic feature extraction from biomimetic tactile sensor: force, contact location and curvature. ROBIO, 2011.
[51] A. Padmanabha, F. Ebert, S. Tian, Calandra, C. Finn, and S. Levine, Omnitact: A multi- directional high-resolution touch sensor. In ICRA, 2020.
[52] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[53] N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. R. Lanckriet, R. Levy, and N. Vas- concelos. A new approach cross-modal multimedia retrieval. In ICM, 2010.
[54] Schulman, S. Levine, Abbeel, M. Jordan, and Moritz, Trust region policy optimization. In ICML, 2015.
[55] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRL, 2020.


Reviews:
[{"review_id": "yPIDxvH_Bt", "reviewer": "Reviewer_arko", "summary": "The authors present ObjectFolder, a dataset containing 100 objects with implicit visual, audio and tactile representations. A novel method is proposed for representing the audio signals implicitly. Once this dataset is built, the implicit functions are evaluated in a series of passive benchmarks measuring 3D reconstruction quality, cross-modal instance recognition/retrieval and grasping stability prediction in simulation. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, I must say, this paper was a joy to read and is very original. The approach is very novel and could potentially be useful for the robotics community which has traditionally focused on explicit data representations with mixed results.\n\nVisionNet and TouchNet seem to have limited contributions since they are mainly inspired from NERF. The representation for AudioNet on the other hand seems to be a novel contribution.\n\nThe biggest concern for the paper is that there is no comparison to explicit methods. For instance, a direct comparison to the network architecture in Calandra et al. [CoRL 2017] would be insightful for the grasp stability experiment. Without this comparison, as a reader I was still wondering why a implicit representation is functionally necessary over an explicit one.\n\nAlso it is challenging to understand if the implicit representation transfers well to real world tasks since the results were only passively tested (i.e. without contact-rich manipulation) or in simulation. \n\nTable 3 is very insightful. It shows that vision and audio are good modalities to capture global information (like the object instance or its category) compared to tactile (which mainly has local geometric information).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I support to accept this paper in its current form. Despite my reservations on the potential impact and widespread adoption of the dataset in the community, there are definitely novel contributions in the idea and architecture.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "SffsXtuU-jS", "reviewer": "Reviewer_pXzD", "summary": "This paper presents a dataset of 100 virtual objects with visual, auditory, and tactile sensory input for learning multi-modal object representations. The dataset is constructed from scanned real-world objects available from ModelHaven, YCB, and Google Scanned Objects. The audio is synthesized through rigid-body sound simulation and touch is simulated with TACTO for emulating a DIGIT sensor. To investigate the usefulness of multimodal representations, the paper presents a model for learning neural representations with three subnetworks – VisionNet, AudioNet, and TouchNet. Evaluations on downstream tasks such as instance recognition, cross-sensory retrieval, and grasping, show that using multimodal representations result in significant performance gains over simple unimodal or bimodal baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- This paper makes an important resource contribution to robotics. Most robot learning methods heavily rely on vision for perception. There has been overwhelming evidence that perception for manipulation is inherently multimodal and to learn robust and dexterous skills, robotic systems need to go beyond just visual perception. \n- Benchmarking and reproducibility are crucial for evaluating progress in various robotic domains. Having a standardized set of objects for multimodal perception, particularly in a simulation, is very beneficial for benchmarking future works in detection, grasping, and dexterous manipulation. To the best of my knowledge, this is the first benchmark that provides all three modalities – vision, audio, and tactile, in a simulated setup.\n- The evaluations on downstream tasks are insightful. In all cases from detection to grasping, the multimodal representations outperform unimodal and bimodal baselines. \n\nWeaknesses: \n- One concern, although relatively minor, is the quality of the audio and tactile simulation and their fidelity to real-world sensors. While the visual rendering of scanned objects is near photorealistic, the audio simulation uses rigid-body simulation to emulate audio sensors. For this simulation, each object is modelled with a single material since individual parts of the scanned objects are unknown. This could be an issue for some objects, for example, striking a hammer on the plastic handle might produce a different sound than striking it on the metallic head. This might also explain why the Audio-only baseline in Table 2 performs so well. Similarly, the simulated tactile sensor uses the geometric structure of scanned objects which could have some imperfections and sharp edges from 3D reconstruction. Ideally, both these issues can be addressed by evaluating the simulated sound and tactile inputs against real-world sensor data. This can be done by picking a few objects from the dataset, collecting data with real-world objects, and comparing it against the simulated audio and tactile sensors.\n- Another concern, noticeable in the supplementary video, is that the dataset contains a lot of cups or cup-like objects of cylindrical shapes. Was this a specific design choice? Or a symptom of what was available/feasible for robot grasping? Since the audio and tactile inputs are automatically simulated, is there anything preventing the dataset from containing more than 100 objects? Perhaps all of Google Scanned Objects and ModelHaven objects? Benchmarks need to be careful not to induce biases in the dataset that might result in models overfitting or exploiting biased data without learning how to solve the task. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper makes an important resource contribution with a multimodal dataset of virtual objects. Experimental evaluations on various downstream robotics tasks indicate that multimodal representations are crucial for going the last mile in detection, grasping, and related tasks. \n\n**Post Rebuttal**  \nI am happy with the authors' response and followup experiments, so I am keeping my original score. I'd encourage the authors to add more diverse objects to the dataset, if possible.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "Minor comments:\n- From the supplementary video, the center-of-mass (COM) of Google Scanned Objects seem to be incorrect, leading to wobbly objects that never fall down. This is a known issue with Google Scanned Objects where the COM is at the bottom of the object rather than it’s geometric center or true center. Accordingly, you probably need to re-run the grasping experiments with the corrected COMs.\n- How many training images were used to train the NeRF? How long does it take to train a NeRF per object? \n"}}, {"review_id": "D5iOG27ybOV", "reviewer": "Reviewer_BbKQ", "summary": "The paper aimed to tackle the problem that Multisensory object-centric perception is limited by the small set of objects available. It presented OBJECTFOLDER, a dataset of 100 virtualized objects with implicit visual, auditory, and tactile representations. The usefulness of the proposed dataset was evaluated on an array of benchmark for both perception and control.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- For robotic applications, it is very expensive to obtain large-scale multisensory dataset, especially auditory and tactile data. The limitations have long been noticed. This paper proposed a novel method to obtain visual, auditory, and tactile sensory data of objects from virtual environments, which is well motivated.\n\n- The paper introduced OBJECTFOLDER, a dataset of 00 virtualized objects that makes multisensory learning with vision, audio, and touch easily accessible to the research community.\n\n- OBJECTFOLDER employs a uniform, object-centric, and implicit representation for each object’s visual textures, acoustic simulations, and tactile readings, making the dataset flexible to use and easy to share.\n\nWeaknesses:\n- Although the dataset contains100 object-centric visual, auditory, and tactile sensory data, the reviewer has doubts that whether such dataset can well represent auditory, and tactile attributes of the real objects. This should be discussed appropriately.\n\n- The paper mentioned that the rendered images, impact sounds, and touch readings well match the ground-truth simulations. If only based on the results in Figure 4, this conclusion may be too arbitrary. It needs more experimental analysis to support this conclusion. \n\n- The paper claims the contribution that all objects in the proposed dataset are platform agnostic, which may not be a valid contribution. As a testbed, it should be the basic requirement.\n\n- The paper adopted a variety of benchmark tasks to demonstrate the validity of the data set, which is unconvincing. The paper should verify whether the dataset is valid for robot perception and learning task in real-life scenarios. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the method of obtaining visual, auditory, and tactile sensory data of objects from virtual environments is helpful for robotic multimodal learning, the presented experimental analysis did not effectively demonstrate the contribution of the paper. The paper needs to be further modified by adding additional experiments to improve the quality. Therefore, it is not ready for acceptance.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- I have doubts that whether such dataset can well represent auditory, and tactile attributes of the real objects. Therefore, additional experimental analysis need to be presented to show that the multisensory data can well represent object attributes.\n\n- The paper mentioned that the rendered images, impact sounds, and touch readings well match the ground-truth simulations. It needs more experimental analysis to support this conclusion. \n\n- According to the focus of the paper, it needs to redesign the experiments to prove the validity of the dataset. In particular, some experiments need to be added to verify effectiveness of the dataset for robot perception and learning task in real scenarios.\n\n- The dataset is evaluated on four different benchmark tasks. This resulted in that experimental details are ignored and the experimental analysis is not sufficient. I think that focusing on one benchmark task can improve the clarity and completeness of the paper.\n"}}]

Author rebuttal:

Meta-review: This paper proposes a new dataset of 100 synthetic objects with visual, auditory, and tactile sensory features for learning multi-modal object representations. The dataset combines objects from the YCB dataset, the ModelHaven, and the Google Scanned Objects. The benefits of the proposed dataset as a testbed for multisensory perception and control are demonstrated by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.
The reviewers agree that the proposed dataset can be useful for pushing ahead the state of the art in robotic grasping and manipulation. A key issue however is the realism of the dataset, which is not clear from this paper. The authors should demonstrate the benefits of using this dataset on real robotic manipulation tasks. 
Decision: Accept (Poster)