Swoosh! Rattle! Thump! - Actions that Sound
Authors: Dhiraj Gandhi
Abhinav Gupta
Lerrel Pinto
Venue:  (2020)
Keywords: Sound
Action
Audio Representations

Abstract: Truly intelligent agents need to capture the interplay of all their senses to build a rich physical understanding of their world. In robotics, we have seen tremendous progress in using visual and tactile perception; however we have often ignored a key sense: sound. This is primarily due to lack of data that captures the interplay of action and sound. In this work, we perform the first large-scale study of the interactions between sound and robotic action. To do this, we create the largest available sound-action-vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot. By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information. Using this data, we explore the synergies between sound and action, and present three key insights. First, sound is indicative of fine-grained object class information, e.g., sound can differentiate a metal screwdriver from a metal wrench. Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied on the object. Finally, object representations derived from audio embeddings are indicative of implicit physical properties. We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings. 

Under review as a conference paper at ICLR 2020
SWOOSH! RATTLE! THUMP! ACTIONS THAT SOUND
Anonymous authors Paper under double-blind review
ABSTRACT
Truly intelligent agents need capture the interplay of all their senses build rich physical understanding of their world. In robotics, we have seen tremendous progress in using visual and tactile perception; however we have often ignored a key sense: sound. This is primarily due to lack of data that captures the interplay of action and sound. In this work, we perform the first large-scale study of the interactions between sound and robotic action. To do this, we create the largest available sound-action-vee vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot. By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information. Using this data, we explore the synergies between sound and action, and present three key insights. First, sound is indicative e.g., sound differentiate metal screwdriver from metal wrench. Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied on the object. Finally, object representations derived from audio embeddings are indicative of implicit physical properties. We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings.
anee-graind object class information,
INTRODUCTION
Imagine the opening of a champagne bottle! Most vivid imaginations not only capture the celebra- tory visuals but also the distinctive 'pop' sound created by the act. Our world is rich and feeds all of our five senses vision, touch, smell, sound and taste. Of these, the sense of vision, touch and sound play critical role our rich physical understanding of objects and actions. A truly intelligent agent would need to capture the interplay of all the three senses to build physical understanding of the world. In robotics, where the goal is to perform physical task, vision has always played a central role. Vision is used to infer the geometric shape (Kar et al. (2015)), track objects (Xiang et al. (2015)), infer object categories (Krizhevsky et al. (2012)) and even direct control (Levine et al. (2016a)). In recent years, the sense of touch has also received increasing attention for recognition (Schneider et al. (2009)) and feedback control (Murali et al. (2018)). But what about sound? From the squeak of door, to the rustle of dried leaf, sound captures rich object information that is often impercep- tible through visual or force data. Microphones (sound sensors) are also inexpensive and robust; yet we haven't 't seen sound data transform robot learning. There hardly exists any systems, algorithms or datasets that exploit sound as vehicle to build physical understanding. Why is that? Why does sound appear be second-class citizen among perceptual faculties?
The key reason lies at the heart of sound generation. Sound generated through an interaction, say robot striking an object, depends on the impact of the strike, the structure of the object, and even the location of the microphone. This intricate interplay that generates rich data, also makes it difficult to extract information that is useful for robotics. Although recent work has used sound determine the amount of granular material in container (Clarke et al. (2018)), we believe there lies much more information in the sound of interactions. But what sort of information can be extracted from this sound?
In this paper, we explore the synergy between sound and action to gain insight into what sound can be used for. To begin this exploration will first need large and diverse dataset that contains both sound and action data. However, most existing sound datasets do not contain information about action, while most action datasets do not contain information about sound. To solve this, we create

---

Under review as a conference paper at ICLR 2020
Action data
Tilt-Bot
Visual data
Audio data
Figure 1: Using Tilt-Bot, collect 15,000 interactions 60 different objects by tilting them in a tray. When sufficiently tilted, the object slides across the tray and hits the walls of the tray. This generates sound, which is captured by four contact microphones mounted on each side of the tray. An overhead camera records visual (RGB+Depth) information, while the robotic arm applies the tilting actions through end-effector rotations.
the largest sound-actin suund-action-viionn dataset available with 15,000 interactions on over 60 objects with Tilt-Bot robot Figure 1. Each object is placed in tray mounted on a robot that is tilted with a random action until the object hits the walls of the tray and make a sound. This setup allows us robustly collect sound and action data over diverse set of objects. But how is this data useful? Through Tilt-Bot's data, we present three key insights about the role sound in action.
The first insight is that sound is indicative of fine-grained object information. This implies that just from the sound an object makes, a learned model can identify the object with 79.2% accuracy from set of diverse 60 objects, which includes 30 YCB objects (Calli al. (2015)). Our second insight is that sound is indicative of action. This implies that just from hearing the sound of an object, a learned model can predict what action was applied the object. On a set of 30 previously unseen objects, we achieve 0.027 MSE error which is 42% better than learning from only visual inputs. Our final insight is that sound is indicative of physical properties of object. This implies that just from hearing the sound an object makes, a learned model can infer the implicit physical properties of the object. To test this implicit physics, we show that learned andio-conditioned forward model achieves LI error of 093 on previously unseen objects, which is 24% lesser than forward models trained using visual information. This further indicates that audio embeddings, generated from previous interaction, can capture information about the physics of an object significantly better than visual embeddings. One could envision using these features to learn policies that first interact to create sound and then the inferred audio embeddings perform actions (Zhou al. (2019)).
In summary. we present three key contributions in this paper: (a) we create the largest sound-action vision robotics dataset; (b) demonstrate that can perform fine grained object recognition using only sound; and (c) we show that sound is indicative of action, both for post-interaction pre- diction, and pre-interaction forward modeling. Tilt-Bot's sound-action-visior data, along with audio embeddings can be accessed here: http::/sttt..goooeecco//wwww///t
2 THE TILT-BOT SOUND DATASET
To study the relationship between sound and actions, first need to create a dataset with sound and action. In this section, we describe our data collection setup and other design decisions.
The Tilt-Bot Setup: A framework collect large-scale data needs three key abilities: (a) to pre- cisely control the actions; (b) to be able to interact with a diverse set of objects; (c) to record rich and diverse sound; and (d) requires little to no manual resets. To do this, we present Tilt-Bot (Figure l). Tilt-Bot is robotic tray mounted on Sawyer robot's end-effector. This allows to precisely control the movements of the tray by applying rotational and translational actions on objects inside it. The tray has dimensions of 30 x 30 em and a payload of Kg allowing to place large range of common day objects in it. To collect audio information, four contact microphones (mic) are at- tached the four sides of the tray. This allows for the creation of rich audio information from the
2

---

Under review as a conference paper at ICLR 2020
Figure 2: Here are 12 interactions collected using Tilt-Bot. Each interaction is visualized as three images. The left image shows the visual observation of the object before the action is applied along with the applied action. The middle image shows the middle image shows the visual observation after the interaction, while the right image shows the STFT representation of audio generated by of the microphones.
interactions of objects with each other and the tray. To collect visual information, Intel Realsense Camera (cam) is mounted on the top of the tray to give RGB and Depth information of the object in the tray. Our current setup allows to collect four-channel audio 400100Hz, RGB and Depth at 6Hz, and tray state information (rotation and translation) at 100Hz. Rotational and translational action commands can be sent at 100Hz.
Data Collection Procedure: Our dataset consists of sound-action-vision data on 60 objects; 30 of which belong the YCB object dataset (Calli et al. (2015)), and 30 are common household objects. Details of these objects can be found in Appendix A. For each object, data is collected by first placing in the center of the tray. Then, Tilt-Bot applies randomly generated rotational actions to the object for 1 hour. We do not apply translational action since notice minimal motion of the object with it. The rotational actions cause the tray to tilt and make the object slide and hit the walls of the tray. The sound from the four microphones, along with the visual data are continually recorded. Furthermore, using a simple background subtraction technique (Zivkovic (2004)), can track the location of the object it collides with the walls of the tray. For every contact made with the tray's wall, which is detected by peaks in the audio stream, we segment four second interaction centered around this contact. This amounts around 15000 interactions over 60 robotic hours of data collection. Each of these interactions contain the sound, the RGB+Depth, and the tracking location of the object during the interaction. Examples of the data can be seen in Figure 2. All of our data and pre-procsssing will be open-sourced, and can be accessed on our website: https//sites.googl..oom/// wtt..///ewww/e//200-0oooo/o/
LEARNING WITH AUDIO
To understand and study the synergies between sound and action, we focus on three broad categories of learning tasks: (a) fine-grained classification (or instance recognition). (b) inverse- modell learn- ing (or action regression). and (c) forward-mode learning In this section, we will describe our experiments along with insights to better understand the role of sound with action in the context of learning.
3.1 PROCESSING AUDIO DATA
Before using audio data for learning, we first need to convert into a canonical form. Since we will audio in conjunction with images for several experiments, build the representation
3

---

Under review as a conference paper at ICLR 2020
Object embedding
ea
Audio embed
Audio embed
e
Selvers
Audio
Class
Interaction
Gen. audio
Image
Action
eI
(og Bry
Image embed
Forward model
(ry -i
(To yol
(a) Training the audio embedding
(b) Audio embedding for forward models
Figure 3: To train audio embeddings (a), we perform multi-task learning for instance recognition (top) and action regression (bottom). Once the embedding network is trained, we can use the ex- tracted audio embeddings as object features for downstream tasks like forward model learning (b).
proposed by Zhao et al. (2018). Here the key idea is to convert the high dimensional raw audio (705600 for 4 second audio recorded at 44.1 4Hz for 4 audio channels) smaller dimensional image. This is first done by subsampling each audio channel from 44. 14.1KM KHz to 1KH Then, a Short-time Fourier transform (STFT) (Daubechies (1990)) with a FFT window size of 510 and hop length of 128 is applied on the subsampled and clipped audio data. For each channel this results in 64 x 64 representation. Stacking the 4 channel audio, we get apply a log transformation and clip the representation to between [-5,5] 5]. This representation allows us treat audio as an image and now effectively rum 2D convolutions on audio data, which can capture the temporal correlations from single audio channel along with the correlations between multiple audio channels. Visualization of this representation can be seen in Figure 2, where the first. three channels of audio data (64 x 64 x 3) are converted an RGB image.
64 x representation. We further
3.2 FINE-GRAINED OBJECT CLASSIFICATION
Classically, the goal of recognition to identify which object is being perceived. This task is generally done using visual images input, and is done to test the robustness of visual feature extractors. In our case, we use this task to study what type of object-centric information is contained first create training set with 80% of the
in sound. For the 60 objects in TiltBot dataset data and testing set with the remaining 20%. Then, we train simple CNN (Krizhevsky et al. (2012)), that only takes the audio information input and outputs the instance label of the object that generated the sound. This architecture is similar to top part of Figure 3(a).
On our heldout testing set, this trained model achieves a classification accuracy of 76.1%. Note that a random classifier gets 1.67% accuracy. This shows that audio data contains fine-grained information about objects. Although Owens et al. (2016) demonstrate that audio information can be used to classify broad categories like wood, metal etc., our results show for the first time (to our knowledge) that audio information generated through action gives instance-level information like screwdriver, scissor, tennis ball etc.. To further understand what information sound gives we study the top classification errors of our model. modes of errors. The first is if instances only differ visually. For example, green cube cannot be distinguished from a blue cube solely from the sound information. The second error mode is the generated sound is too soft. If the action causes the object to only move a little and not make too much sound, information about the object is masked away and causes classification errors.
Figure we see that there are two main
3.3 INVERSE-MODEL LEARNING
The goal of learning inverse models is to identify what action applied, given observations be- fore and after the action. From a biological perspective, learning inverse-mout models implies an under- standing of cause and effect, and often necessary for efficient motor-leaming (Wolpert & Kawato
4

---

Under review as a conference paper at ICLR 2020
GT: cube
GT: rube Rlur cale
Vellow cabe Screwdriver Ch Med. clamp
Figure 4: Here we see the top classification errors made by our instance recognition model. There are two key failure modes: (a) misclassification between similar objects that have different visual properties like the cubes, and (b) when the sound generated is too soft like the misclassification of the screwdriver.
Train objects
Test objects
0.0
0.05
0.1
0.2
1.0
0.0
0.1
0.2
1.0
Image
Class. (1)
0.738
0.780
0.770
0.786
0.027
N/A
N/A
N/A
N/A
N/A
Reg. (L)
0.395
0.024
0.022
0,014
0.008
0.352
0.027
0.020
0.027
0.043
Table
Classification and Regression performance across different methods on the Tilt-Bot dataset.
Figure 5: Visualization of predictions from our inverse model. Here, the red arrow represents the ground truth actions taken by the robot, while the black arrow corresponds the actions predicted by our action-regression model.
(1998)). In the general setting of this problem, model takes input the observations before and af- ter an interaction, and outputs the action applied during the interaction. In our case, we want to study if sound contains cause-effect information about actions. Moreover, since inverse-model learning can be evaluated on previously unseen objects, we can test the generalization of audio features not only objects in training, but novel objects well.
To demonstrate this, we split our TiltBot objects into two sets: set A and set B, where both sets contain 30 objects with 15 objects from the YCB dataset. Using architecture similar to the bottom part of Figure 3(a), an inverse model is trained on set A to regress the action. The input into this inverse model is image of the object before the interaction, and the sound generated during the interaction. Note that the image of the object after the interaction is not given as input. The action that needs be output the 2D projection of the rotation vector on the planar tray surface. We evaluate the performance of the inverse model using normalized (]), mean squared error (MSE), where lower better. Testing this model on held-out set A objects, we get a MSE of 0.008, while random model give MSE of 0.4. If we use the image of the object after the interaction as input instead of audio, we get a MSE of 0.043. This shows that for these physical interactions using audio information is not just better than random, but in fact better than using visual observations. This insight holds true even when tested on previously unseen set B objects. With set B testing, audio inverse models give MSE of 0.027, which indicates some amount of overfitting objects. However, this is significantly better than the 0.047 MSE we get from using purely visual inverse models. Sample evaluations of our inverse model can be in Figure 5.
set A

---

Under review as a conference paper at ICLR 2020
Figure 6: Image retrieval results based on audio embedding. Here, the green box image corresponds to query image and next 3 images in the row (in red) correspond to retrievals.
3.4 MUSKI-TASK AUDIO EMBEDDING LEARNING
In the previous two sections have seen how sound contains information about both fine- tne-grained instances of objects and causal effects of action. But what the right loss function to train an audio embedding that generalizes to multiple downstream tasks? One way would be to train the embedding on the instance recognition task on Tilt-Bot data, while another option would be to train on the inverse-model task. Both of these tasks encode different forms of information, with classification encoding identifiable properties of the object and inverse model encoding the physical properties of the object. Inspired from work in multi-task learning (Caruana (1997); Pinto & Gupta (201 6a)), we take the best of both worlds and train joint embedding that can simultaneously encode both classification and action information.
As seen in Figure 3(a), the audio embedding e is trained jointly using the classification and the inverse model loss according to Liotal (1 Celass + XLene. Note that when X 0, the embed- ding captures only classification information, while X = I captures only inverse-mode information. We report the performance of joint learning on held-out data in Table Here, training is performed on set A objects, while testing is done set A held-out interactions and unseen set B objects. For classification, we find that joint learning improves performance from 73.8% on the 30 set objects to 78.6%. When trained both set A and set B objects, classification performance improves from 76.1%( Section 3.2) to 79.5%. On rnverse-model learning, we notice that joint learning does not improve performance
set A. However, from 0.027 MSE to 0.020 MSF. Again, this performance also much better than learning directly from visual inverse-models 0.043 MSE.
novel set B objects, we see significant improvement
Another way to understand the information captured in our audio embeddings is to look at the top three nearest instance categories given an input object. In Figure we show a few of these object retrievals. Interestingly, these features capture object shapes like matching the long screwdriver to the long butterknife and matching the yellow cube to other colored cubes.
3.5 DOWNSTREAM TASK: FORWARD MODEL PREDICTION
Our previous experiments demonstrate the importance of using audio perception. In this section, we investigate if we can use sound to extract physical properties of an object before physically interacting with it. This use case is inspired from recent work on EPI (Zhou al. (2019)) where probing interactions are used to understand latent factors before implementing the real policy. Here the sound generated through probing interactions would serve latent parameters for representing the object.
To evaluate the use of audio features for downstream tasks, we perform forward prediction (See Figure 3(b)). Here given an object, random interaction is performed on it and a sound is generated from this interaction. The embedding network trained using multi-task learning is then used to ex- tract the audio embedding, which will serve as our object's representation. Then, given this object's representation, we can train forward model that takes as additional input the location of the object and action applied the object, and outputs the location of the object after the interaction. To learn this forward model, the network has to understand the dynamics of the object. Note that the only object specific information is given through the audio embedding.

---

Under review as a conference paper at ICLR 2020
Audio Embeddings
No Audio
0
0.05
0.1
0.2
1.0
ResNet
Oracle
Train Objects
0.225
0.221
0.220
0.222
0.239
0.258
0.206
Test Objects
0.195
0.194
0.193
0.1945
0.195
0.256
0.155
Table 2: Comparison of using audio embeddings versus pre-trained visual embeddings for forward model prediction. Oracle represents training with object class labels as input.
Figure 7: Forward model predictions: Here, green dot on the image represents the ground truth location. Based on the action taken by robot, shown in red arrow initial state of the tray and corresponding object ground truth location, forward model tries to predict object location which is shown as red dot.
As seen in Table 2, we report sigmificant improvements in forward model prediction from 0.258 L1 error using visual features 0.220 L1 error when using the audio embedding feature objects seen during forward model training. This trend continues for novel set B objects, where both the embedding and forward model was trained on set objects. Here we
even larger improvement with visual features giving 0.256 LI error while audio features giving 0.193 LI error. This shows that audio embedding information is better able to capture implicit physics of the object as compared to visual features. Moreover, these features are significantly more robust than visual features and also generalize to previously unseen objects.
4 RELATED WORK
4.1 MOLTI-MODAL LEARNING WITH SOUND
Recently, there has been growing interest erating sound for mute videos, or to localize the part of image that produces sound, or to learn better visual and audio features. Owens al. (2015) collected hundreds of videos of people hitting, scratching, and prodding objects with drumstick. This data was then used to train a recurrent neu- ral network which synthesizes sound from silent videos. We collect audio of interactions between objects and tray. However, instead of relying on humans, which is huge bottleneck for data col- lection, we use a robotic platform to collect data. Aytar et al. (2016) use the natural synchronization between vision and sound to learn acoustic representation using unlabelled two-million videos. Similarly, Arandjelovic & Zisserman (2017a) 7a) looks at raw unconstrained videos to learn visual and audio representations that perform on par with ffute-of-the-ar self-supervised approaches. In sim- ilar spirit, we also learn audio representations, albeit through action, to be used for down stream tasks. Arandjelovic & Zisserman (2017b); Senocak et al. (2018) further explores the audio-visual
sound in conjunction with vision, either for gen-

---

Under review as a conference paper at ICLR 2020
correspondence in videos to localize the object that sounds in image, given the audio signal. Zhao et al. (2018) has taken this idea one step further. Given the audio signal, they separate it into a set of components that represents the sound from each pixel. In contrast these works, we look at obtaining a richer representation of sound by studying its interactions with action.
4.2 LEARNING FORWARD MODELS
Standard model-based methods, based on the estimated physical properties and known laws of physics, tries to calculate sequence of control actions to achieve the goal (Khatib (1987); Mur- ray (2017): Dogar & Srinivasa (2012)). Although this has been widely used for object manipulation tasks in robotics (Cosgun et al. (201 I)), manipulating an unknown object is still a daunting task for such methods. This is mainly due to the difficulties estimating and modeling the novel physical world (Yu et al. (2016)). Given the challenges in predicting the physical properties of novel envi- ronment, Deisenroth & Rasmussen (2011); Gal et al.; Amos et al. (2018); Henaff et al. (2017) try to learn dynamic models based their interactions with the environment. However, when need to use these learned models on previously unseen objects, these models also fail to generalize. This is because they often do not contain object-specifie information. One way to get object-specific infor- mation is to use raw visual observations instead of object state (Agrawal et al. (2016); Hafner et al. (2018); Finn & Levine (2017)). In these methods, given the observation of a scene and an action taken by the agent, visual forward model predicts the future visual observation. These forward models can then be used to plan robotic motions. In our work, we show that instead of using visual information, audio embeddings generated from a previous interaction can be used to improve these forward models.
4.3 MULTI-MODAL DATASETS
Alongside algorithmic developments, large scale datesets have enabled the application of machine learning to solve numerous robotic tasks. Several works like Pinto & Gupta (2016b); Levine et al. (2016b); Agrawal et al. (2016); Gandhi et al. (2017) collect large scale visual robotic data for learn- ing manipulation and navigation skills. Apart from visual data, some works Murali et al. (2018); Pinto al. (2016) have also looked at collecting large-scale tactile data. This tactile or force data can then be used to recover object properties like softness or roughness. Although these datasets contain visual information and action data, they ignore key sensory modality: sound.
Understanding what information can be obtained from sound requires a large-scale sound dataset. Early attempts (Owens al. (2015)) collect sound data by recording people interacting with objects. Although this dataset contains large amounts of sound data, it does not contain information about action. In our work, we show that action information not only helps regularize object classification, but also helps in understanding the implicit physics of objects. Prior to our work, Clarke et al. (2018) has shown that sound information is indeed helpful for state-estimation tasks like measuring the amount of granular material in container. Here, they exploit the mechanical vibrations of granular material and the structure around it for accurate estimation. In our work, instead a single type of object, we collect audio data across 60 different objects. This allows us to learn generalizable audio fcatures that transfer to previously unseen objects on a variety of tasks like action regression and forward-model learning.
CONCLUSION
In this work, we perform one of the first studies on the interactions between sound and action. Through our sound-action-vision dataset collected using our Tilt-Bot robot, we present several in- sights into what information can be extracted from sound. From fine-grained object recognition, to inverse-model learning, we demonstrate that sound can provide valuable information that can be used in downstream motor-control or robotic tasks. In some domains like forward model learning, show that sound infact provides more information that be obtained from visual information alone. We hope that the Tilt-Bot dataset along with our findings inspire future work in the sound- action domain, especially in robotic settings where visual data is hard to obtain.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Intel
realsense d435 rgbd
camera.
https://click intel.com
intelr- oateel---raalsesnsnn--dddr--eeee
html.
http:///www....a.oo.oooo///ooooa dp/B019TW4B20.
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. NIPS, 2016.
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gómez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. arXiv preprint arXiv: 0e04.80044.0/.. 2018.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. CoRR, abs/ I 705,08168, 2017a.
URL http::ttp//aaat org/ aww.////..... 08168.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. CoRR, abs/1712.0665 2017b.
URL httt:://axxi org/ wwwp///2....... 06651.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound represen- tations from unlabeled video. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems 29, pp. 892-900. Curran Associates, Inc., 2016. URL http::/pappeee nips cc/paper/ 6146- -en.dcce----ennnnnn---o--rrrreee htw.:a/aaattttoo--oooommemaaaaea pdf.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dol- lar. Benchmarking in manipulation research: The ych object and model set and benchmarking protocols. arXiv preprint arXiv:1502.03144 2015.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Samuel Clarke, Travers Rhodes, Christopher G. Atkeson, and Oliver Kroemer. Learning audio feedback for estimating amount and flow of granular material. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 529-550. PMLR, 29-31 Oct 2018. URL http::///// / http:////ocoedddeeeeoooo
mllr www...pres//vee /88 cwwckeaaalleeaeocaeocee 8a .tml..
Akansel Cosgun, Tucker Hermans, Victor Emeli, and Mike Stilman. Push planning for object place- ment on cluttered table surfaces. 201 JEEE/RSJ international conference on intelligent robots and systems, pp. 4627 4632. IEEE, 2011.
Ingrid Daubechies. The wavelet transform, time-frequency localization and signal analysis. IEEE fransactions on information theory, 36(5):611-0055, 1990.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp- 465 472,442,22 2011.
Mehmet R Dogar and Siddhartha S Srinivasa. A planning framework for non- non-prehensile manipula- tion under clutter and uncertainty. Autonomous Robots, 33(3):217-236, 2012.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. 2017 IEEE International Conference Robotics and Automation (ICRA). pp. 2786-2793. IEEE, 2017.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models.
Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta. Learning to fly by crashing. CoRR, abs/1704.05588, 2017. URL http: // wwwpaawwwwwww..////////////////. 05588.

---

Under review as a conference paper at ICLR 2020
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1...511,,,,, 2018.
Mikael Henaff, William F Whitney, and Yann LeCun. Model-based planning with discrete and continuous actions, arXiv preprint arXiv:/705.07177 2017.
Abhishek Kar, Shubham Tulsiani, João Carreira, and Jitendra Malik. Category-specifiic object re- construction from a single image. In Computer Vision and Pattern Regognition (CVPR), 2015.
Oussama Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE J. Robotics and Automation, 3:43-53, 1987.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. JMLR, 2016a.
Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. ISER, 2016b.
Adithyavairavan Murali, Yin Li, Dhiraj Gandhi, and Abhinav Gupta. Learning to grasp without seeing. CoRR, abs/ 805.04201, 2018. URL http: / //tp///aaavv////a//ea//eeeeaaeva wwwp:/i.wrg/ab////0000 04201.
Richard M Murray. A mathematical introduction robotic manipulation. CRC press, 2017.
Andrew Owens, Phillip Isola, Josh H. McDermott, Antonio Torralba, Edward H. Adelson, and William Freeman. Visually indicated sounds. CoRR, abs/1512.08512,, 2015. URL http:
//arxiv. org.
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2403-2413, 2016.
Lerrel Pinto and Abhinav Gupta. Learning push by grasping: Using multiple tasks for effective learning. arXiv preprint 1609. 2016a.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. ICRA, 2016b.
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. ECCV, 2016.
Alexander Schneider, Jürgen Sturm, Cyrill Stachniss, Marco Reisert, Hans Burkhardt, and Wolfram Burgard. Object identification with tactile sensors using bag-of- bag-of-features. In IROS, volume 9, pp. 243-248, 2009.
Arda Senocak, Tae-Hyun Oh, Jun-Sik Kim, Ming- HinggHsuann Yang, and In So Kweon. Learning to localize sound source in visual scenes. CoRR, tbss/103.00389, 2018. URL http: org/abs: hrg/abs/11n /1//3/0033 03849.
Daniel M Wolpert and Mitsuo Kawato. Multiple paired forward and inverse models for motor control. Neural networks, 11(7-8):111--3322, 1998.
Yu Xiang, Alexandre Alahi, and Silvio Savarese. Learning to track: Online multi-object tracking by decision making, In International Conference Computer Vision (ICCV), pp. 4705-4713, 2015.
Kuan-Ting Ting Yu, Maria Bauza, Nima Fazeli, and Alberto Rodriguez. More than million ways to be experimental dataset of planar pushing. In 2016 IEEE/RSJ international
pushed. a conference on intelligent robots and systems (IROS), pp. 30-37. IEEE, 2016.
Hang Zhao, Chuang Gan, Andrew Rouditchenko. Carl Vondrick, Josh McDermott, and Antonio Tor- ralba. The sound of pixels. In The European Conference on Computer Vision (ECCV), September 2018.
10

---

Under review as a conference paper at ICLR 2020
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. ICLR, 2019.
Zoran Zivkovic. Improved adaptive gaussian mixture model for background subtraction. In Proceed- ings of the 17th International Conference on Pattern Recognition, 2004, ICPR 2004., volume 2,
pp. 28-31. IEEE, 2004.

---

Under review as a conference paper at ICLR 2020
APPENDIX A DATASET DETAILS
Figure 8 shows image of objects for which the data was collected using Titt-Bot. Moreover, Table and Table 4 shows the number of interactions collected for each object in set A (seen objects) and set B (novel objects) respectively.
800
Figure 8: Image of objects for which data was collected using Tott-Bot.
Object Name
Number of interactions
pudding box
141
baseball blue block
365
197
salt cylinder
309
tennis ball
331
whiteboard spray
285
toothpaste box
173
chain
115
strawberry
323
flat screwdriver
324
medium clamp
172
spoon
240
rubiks cube
247
glass
264
red block
172
scissors
216
pear
336
plum
331
robot dog with worker
320
yellow block
176
phillips screwdriver
289
acrylic bottle
365
brown green toy
171
mug + large clamp baseball
612
four cubes
352
green block
197
mug
239
tuna fish pudding box + apple
444
adjustable wrench
217
green chip clip
131
Table 3: Train Data
12

---

Under review as a conference paper at ICLR 2020
Object Name
Number of interactions
timer
179
tuna fish can stanley screwdriver
197
332
blue fidget
297
large gift box rubiks cube jumbled
117
131
strawberry
356
wd 40
200
lemon
389
large clamp
232
small marker
307
apple
229
chocolate mint bar
105
shiny toy gun blue nerf gun blue toy car green basketball nine hole peg test old spice swagger
233
236
122
414
245
199
wine glass
432
fork
249
toy elephant
180
pink heel
124
rubiks cube
231
sugar box
185
bowl
206
knife
226
foam brick
71
soft cube
67
black tape
100
Table Test Data
13


Reviews:
[{"review_id": "rJlDUEniFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the interesting connections between action and sound, by building a sound-action-vision dataset with a tilt-bot. This is a good paper overall, I appreciate the efforts on the dataset, and this direction of research is worth pursuing. \n\nRegarding experiments, I like the way it is set up, especially the four microphones, and the action space of the robot. I\n\nA couple of questions:\n(1) In the inverse model learning, Fig 3(a) bottom, why are images used as input as well? Don't we want to predict action purely from sound?\n(2) In forward model prediction, how are the ground truth locations defined and labeled? Is it the center of mass, and annotated by humans? More details on this experiment will help.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skxm-jETFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the role of audio in object and action perception, as well as how auditory information can help learning forward and inverse dynamics models. To do this, the authors built a 'tilt-bot', which tilts a box and the object within to collect data (sound & vision) of object interactions. The authors then tested how audio embeddings help object recognition and forward model prediction. \n\nThe idea presented in this paper is quite interesting. However, there are no significant technical innovations, the experimental evaluations are quite limited, and the writing can be improved. My overall recommendation is weak reject.\n\nThe problem of integrating audio for perception is interesting and has been quite widely explored; however, this paper extends the setup to also explore the effect of audios on dynamics modeling. This is relatively new and may lead to many potential future developments in this direction.\n\nTechnically, however, this paper mostly builds on existing technicals on learning forward and inverse models, except that the input is now audio in addition to video. The experimental results are also very limited. They are restricted to a single domain, a fixed collection of objects, and there are no comparisons with published, SOTA algorithms. There are also no results on downstream tasks such as robot manipulation.\n\nI also wonder how the authors think of the related work from Zhang et al: http://sound.csail.mit.edu/ , as they've also studied the effect of auditory and visual data in shape and material recognition.\n\nThe writing can be improved. Currently, the model and results are in the same section and mixed together. It'd prefer to separate them. There are a number of typos (incorrect spacing, etc.), especially in Section 3.4. Please double check.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyxux2VH9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents audio-visual object classification and motion prediction work on a novel dataset of 60 different objects rolling around in a bin tilted to and fro by a robot, with video and 4-channel audio recordings of the object impacts.   The data is rather novel, is large enough to do ML (around 17 hours of eventful audio/video) and is to be publicly released.  The model architectures  are not of theoretical novelty.   However, the experiments are somewhat interesting.  It was found that the audio contains significant object classification information.  The audio was also good for predicting the trajectory of the object.  This might not be surprising since the microphones are geometrically arranged and may contain directional information along with information about velocity and/or distance traveled.   Overall the experiments are rather thin with only a few experimental results.   A more thorough undertaking might be expected for ICLR papers, with more novel theoretical development and more extensive experiments.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates using sound to improve classification, motion prediction, and representation learning all from data generated by a real robot.

All the reviewers were intrigued by the work. The paper provides experiments on real robots (never a small task), and a data-set for the community, and a sequence of illustrative experiments. Because the paper combines existing techniques, its main contribution is the empirical demonstrations of the utility of using sound. Overall, it was not quite enough for the reviewers. The main issues were: (1) motion prediction is perhaps expected given the physical setup, (2) lack of comparison with other approaches, (3) lack of diversity in the demonstrations (10 objects, one domain).

The authors added two new experiments with a different setup, further demonstrating their claims. In addition the authors highlighted that the novelty of this task means there are no clear baselines (to which r3 agreed). The new experiments are briefly described in the response (and visuals on a website), but the authors did not update the paper. The new experiments could potentially significantly strength the paper. However, the terse description in the response and the supplied visuals made it difficult for the reviewers to judge their contribution.

Overall, this is certainly a very interesting direction. The results on real world data demonstrate promise, even if they are not the benchmarking style the community is used too.   
Decision: Reject