Swoosh! Rattle! Thump! - Actions that Sound

1 INTRODUCTION . Imagine the opening of a champagne bottle ! Most vivid imaginations not only capture the celebratory visuals but also the distinctive ‘ pop ’ sound created by the act . Our world is rich and feeds all of our five senses – vision , touch , smell , sound and taste . Of these , the sense of vision , touch and sound play a critical role in our rich physical understanding of objects and actions . A truly intelligent agent would need to capture the interplay of all the three senses to build a physical understanding of the world . In robotics , where the goal is to perform physical task , vision has always played a central role . Vision is used to infer the geometric shape ( Kar et al . ( 2015 ) ) , track objects ( Xiang et al . ( 2015 ) ) , infer object categories ( Krizhevsky et al . ( 2012 ) ) and even direct control ( Levine et al . ( 2016a ) ) . In recent years , the sense of touch has also received increasing attention for recognition ( Schneider et al . ( 2009 ) ) and feedback control ( Murali et al . ( 2018 ) ) . But what about sound ? From the squeak of a door , to the rustle of a dried leaf , sound captures rich object information that is often imperceptible through visual or force data . Microphones ( sound sensors ) are also inexpensive and robust ; yet we haven ’ t seen sound data transform robot learning . There hardly exists any systems , algorithms or datasets that exploit sound as a vehicle to build physical understanding . Why is that ? Why does sound appear to be second-class citizen among perceptual faculties ? The key reason lies at the heart of sound generation . Sound generated through an interaction , say a robot striking an object , depends on the impact of the strike , the structure of the object , and even the location of the microphone . This intricate interplay that generates rich data , also makes it difficult to extract information that is useful for robotics . Although recent work has used sound to determine the amount of granular material in a container ( Clarke et al . ( 2018 ) ) , we believe there lies much more information in the sound of interactions . But what sort of information can be extracted from this sound ? In this paper , we explore the synergy between sound and action to gain insight into what sound can be used for . To begin this exploration we will first need a large and diverse dataset that contains both sound and action data . However , most existing sound datasets do not contain information about action , while most action datasets do not contain information about sound . To solve this , we create the largest sound-action-vision dataset available with 15,000 interactions on over 60 objects with our Tilt-Bot robot Figure 1 . Each object is placed in a tray mounted on a robot arm that is tilted with a random action until the object hits the walls of the tray and make a sound . This setup allows us to robustly collect sound and action data over a diverse set of objects . But how is this data useful ? Through Tilt-Bot ’ s data , we present three key insights about the role of sound in action . The first insight is that sound is indicative of fine-grained object information . This implies that just from the sound an object makes , a learned model can identify the object with 79.2 % accuracy from set of diverse 60 objects , which includes 30 YCB objects ( Calli et al . ( 2015 ) ) . Our second insight is that sound is indicative of action . This implies that just from hearing the sound of an object , a learned model can predict what action was applied to the object . On a set of 30 previously unseen objects , we achieve a 0.027 MSE error which is 42 % better than learning from only visual inputs . Our final insight is that sound is indicative of physical properties of object . This implies that just from hearing the sound an object makes , a learned model can infer the implicit physical properties of the object . To test this implicit physics , we show that a learned audio-conditioned forward model achieves a L1 error of 0.193 on previously unseen objects , which is 24 % lesser than forward models trained using visual information . This further indicates that audio embeddings , generated from a previous interaction , can capture information about the physics of an object significantly better than visual embeddings . One could envision using these features to learn policies that first interact to create sound and then use the inferred audio embeddings to perform actions ( Zhou et al . ( 2019 ) ) . In summary , we present three key contributions in this paper : ( a ) we create the largest sound-actionvision robotics dataset ; ( b ) we demonstrate that we can perform fine grained object recognition using only sound ; and ( c ) we show that sound is indicative of action , both for post-interaction prediction , and pre-interaction forward modeling . Tilt-Bot ’ s sound-action-vision data , along with audio embeddings can be accessed here : https : //sites.google.com/view/iclr2020-sound-action . 2 THE TILT-BOT SOUND DATASET . To study the relationship between sound and actions , we first need to create a dataset with sound and action . In this section , we describe our data collection setup and other design decisions . The Tilt-Bot Setup : A framework to collect large-scale data needs three key abilities : ( a ) to precisely control the actions ; ( b ) to be able to interact with a diverse set of objects ; ( c ) to record rich and diverse sound ; and ( d ) requires little to no manual resets . To do this , we present Tilt-Bot ( Figure 1 ) . Tilt-Bot is a robotic tray mounted on a Sawyer robot ’ s end-effector . This allows us to precisely control the movements of the tray by applying rotational and translational actions on objects inside it . The tray has dimensions of 30× 30 cm and a payload of 1 Kg allowing us to place a large range of common day objects in it . To collect audio information , four contact microphones ( mic ) are attached on the four sides of the tray . This allows for the creation of rich audio information from the interactions of objects with each other and the tray . To collect visual information , an Intel Realsense Camera ( cam ) is mounted on the top of the tray to give RGB and Depth information of the object in the tray . Our current setup allows us to collect four-channel audio at 44,100Hz , RGB and Depth at 6Hz , and tray state information ( rotation and translation ) at 100Hz . Rotational and translational action commands can be sent at 100Hz . Data Collection Procedure : Our dataset consists of sound-action-vision data on 60 objects ; 30 of which belong to the YCB object dataset ( Calli et al . ( 2015 ) ) , and 30 are common household objects . Details of these objects can be found in Appendix A . For each object , data is collected by first placing it in the center of the tray . Then , Tilt-Bot applies randomly generated rotational actions to the object for 1 hour . We do not apply translational action since we notice minimal motion of the object with it . The rotational actions cause the tray to tilt and make the object slide and hit the walls of the tray . The sound from the four microphones , along with the visual data are continually recorded . Furthermore , using a simple background subtraction technique ( Zivkovic ( 2004 ) ) , we can track the location of the object as it collides with the walls of the tray . For every contact made with the tray ’ s wall , which is detected by peaks in the audio stream , we segment a four second interaction centered around this contact . This amounts to around 15000 interactions over 60 robotic hours of data collection . Each of these interactions contain the sound , the RGB+Depth , and the tracking location of the object during the interaction . Examples of the data can be seen in Figure 2 . All of our data and pre-processing will be open-sourced , and can be accessed on our website : https : //sites.google.com/view/iclr2020-sound-action . 3 LEARNING WITH AUDIO . To understand and study the synergies between sound and action , we focus on three broad categories of learning tasks : ( a ) fine-grained classification ( or instance recognition ) , ( b ) inverse-model learning ( or action regression ) , and ( c ) forward-model learning . In this section , we will describe our experiments along with insights to better understand the role of sound with action in the context of learning . 3.1 PROCESSING AUDIO DATA . Before using audio data for learning , we first need to convert it into a canonical form . Since we will use audio in conjunction with images for several experiments , we build on the representation proposed by Zhao et al . ( 2018 ) . Here the key idea is to convert the high dimensional raw audio ( 705600 for a 4 second audio recorded at 44.1KHz for 4 audio channels ) to a smaller dimensional image . This is first done by subsampling each audio channel from 44.1KHz to 11KHz . Then , a Short-time Fourier transform ( STFT ) ( Daubechies ( 1990 ) ) with a FFT window size of 510 and hop length of 128 is applied on the subsampled and clipped audio data . For each channel this results in a 64×64 representation . Stacking the 4 channel audio , we get a 64×64×4 representation . We further apply a log transformation and clip the representation to between [ −5 , 5 ] . This representation allows us to treat audio as an image and now effectively run 2D convolutions on audio data , which can capture the temporal correlations from a single audio channel along with the correlations between multiple audio channels . Visualization of this representation can be seen in Figure 2 , where the first three channels of audio data ( 64× 64× 3 ) are converted to an RGB image . 3.2 FINE-GRAINED OBJECT CLASSIFICATION . Classically , the goal of recognition is to identify which object is being perceived . This task is generally done using visual images as input , and is done to test the robustness of visual feature extractors . In our case , we use this task to study what type of object-centric information is contained in sound . For the 60 objects in our TiltBot dataset we first create a training set with 80 % of the data and a testing set with the remaining 20 % . Then , we train a simple CNN ( Krizhevsky et al . ( 2012 ) ) , that only takes the audio information as input and outputs the instance label of the object that generated the sound . This architecture is similar to top part of Figure 3 ( a ) . On our heldout testing set , this trained model achieves a classification accuracy of 76.1 % . Note that a random classifier gets a 1.67 % accuracy . This shows that audio data contains fine-grained information about objects . Although Owens et al . ( 2016 ) demonstrate that audio information can be used to classify broad categories like wood , metal etc. , our results show for the first time ( to our knowledge ) that audio information generated through action gives instance-level information like screwdriver , scissor , tennis ball etc .. To further understand what information sound gives us , we study the top classification errors of our model . In Figure 4 we see that there are two main modes of errors . The first is if instances only differ visually . For example , a green cube can not be distinguished from a blue cube solely from the sound information . The second error mode is the generated sound is too soft . If the action causes the object to only move a little and not make too much sound , information about the object is masked away and causes classification errors .

Summary: This paper presents audio-visual object classification and motion prediction work on a novel dataset of 60 different objects rolling around in a bin tilted to and fro by a robot, with video and 4-channel audio recordings of the object impacts.   The data is rather novel, is large enough to do ML (around 17 hours of eventful audio/video) and is to be publicly released.  The model architectures  are not of theoretical novelty.   However, the experiments are somewhat interesting.  It was found that the audio contains significant object classification information.  The audio was also good for predicting the trajectory of the object.  This might not be surprising since the microphones are geometrically arranged and may contain directional information along with information about velocity and/or distance traveled.   Overall the experiments are rather thin with only a few experimental results.   A more thorough undertaking might be expected for ICLR papers, with more novel theoretical development and more extensive experiments.