That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation
Authors: Abitha Thankaraj
Lerrel Pinto
Venue: CoRL 2023 Poster (2023)
Keywords: Dynamic manipulation
Self supervised learning
Audio

Abstract: Learning to produce contact-rich, dynamic behaviors from raw sensory data has been a longstanding challenge in robotics. Prominent approaches primarily focus on using visual and tactile sensing. However, pure vision often fails to capture high-frequency interaction, while current tactile sensors can be too delicate for large-scale data collection. In this work, we propose a data-centric approach to dynamic manipulation that uses an often ignored source of information -- sound. We first collect a dataset of 25k interaction-sound pairs across five dynamic tasks using contact microphones. Then, given this data, we leverage self-supervised learning to accelerate behavior prediction from sound. Our experiments indicate that this self-supervised `pretraining' is crucial to achieving high performance, with a 34.5% lower MSE than plain supervised learning and a 54.3% lower MSE over visual training. Importantly, we find that when asked to generate desired sound profiles, online rollouts of our models on a UR10 robot can produce dynamic behavior that achieves an average of 11.5% improvement over supervised learning on audio similarity metrics. Videos and audio data are best seen on our project website: aurl-anon.github.io

That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation
Abitha Thankaraj New York University abitha@nyu. edu
Lerrel Pinto New York University lerrel0ca nyu. edu
Abstract: Learning to produce contact-rich, dynamic behaviors from raw sen- sory data has been longstanding challenge in robotics. Prominent approaches primarily focus on using visual and tactile sensing. However, pure vision often fails to capture high-frequency interaction, while current tactile sensors can too delicate for large-scale data collection. In this work, we propose a data-centric approach to dynamic manipulation that uses an often ignored source of informa- tion sound. We first collect dataset of 25k interaction-sound pairs across five dynamic tasks using contact microphones. Then, given this data, we leverage self-supervised learning to accelerate behavior prediction from sound. Our ex- periments indicate that this supervipe 'pretraining' is crucial to achieving high performance, with a 34.5% lower MSE than plain supervised learning and a 54.3% lower MSE over visual training. Importantly, we find that when asked to generate desired sound profiles, online rollouts of our models on UR10 robot can produce dynamic behavior that achieves an average of 11.5% improvement over supervised learning audio similarity metries. Videos and audio data are best seen on our project website: https://aurl-ang github.io/..
Keywords: Dynamic manipulation, Self supervised learning, Audio
Introduction
Imagine learning to strike a tennis ball. How can you tell whether your shot is getting better? Perhaps the most distinctive feeling is the crisp, springy boom - that just sounds right. It is not just playing tennis, for which audition provides rich and direct signal of success; for example consider everyday tasks like unlocking a door, cracking an egg, or swatting a fly. Recent works in neuroscience [1. 2, 3] have found that sound enhances motor learning of complex motor skills such rowing, where being able to listen to the sound of the rowing machine during learning improves person's eventual rowing ability.
In the context of robotics, the learning of motor skills has often been centered around using visual observations as input [4, 5, 6]. While this has enabled impressive success in a variety of quasi-static robotics problems, visual observations are often insufficient in dynamic, contact-rich manipulation problems. One reason for this is that while visual data contains high amounts of spatial information. it misses out on high-frequency temporal information that is crucial for dynamic tasks. Further, dynamic contact can often be challenging to perceive using only vision. To address this challenge, we will need to embrace other forms of sensory supervision that can provide this information.
Using audio holds promise in remedying the temporal information gap present in vision, and has been explored several prior works. For example, sound has been shown to estimate the volume of granular material in container [7], estimate the height of the liquid in pouring [8], identify object properties [9], and be used alongside vision to locfe [10]. Simultancously. several works in the computer vision community have shown that audio can even be used to extract visual information
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Input sound
Dynamic behavior generated from AuRL
Output sound
Rotting
Shaking
Swatting
Homoontal
Strking
Striking
Verical
Figure 1: Given desired sounds as input (left column), AURL can generate dynamic, contact-rich behaviors on a UR10 robot (middle) that outputs sound similar to the input (right column). For tasks using multiple microphones, for several of these dynamic tasks, the visual changes over time are difficult to perceive, while the sound generated by the movement gives concise signal.
visualize the waveform only from single microphone. Note that
in scene [11, 12]. The foundation these prior works have built begs the question, can audio also be leveraged to learn contact-rich dynamic manipulation?
In this work, present AURL,, new learning-base based framework for auditory perception in robots that generate sound while interacting with their environment. AURL takes multi-channel sound data as input and outputs parameters that control dynamic behavior. We obtain our sound data from contact microphones placed on and around the robot. Directly reasoning over raw audio data would require gathering large quantities of training data, which limits the use of high-capacity deep networks. To address this, we use self-supervised learning methods to learn low-dimensional rep- resentations from audio. These representations form the backbone of AURL and enable efficient learning of dynamic skills with datasets.
We experimentally evaluate AURL on five contact-rich manipulation tasks including swatting fly swatter, striking a box, and rattling child's toy on a UR10 robot (See Fig. 1). Across all tasks, AURL can predict parameters for dynamic behavior from audio with lower errors than our baselines. Furthermore, when given desired sounds to emulate, the dynamic behavior produced by AuRL can generate sounds with high similarity to the desired ones. On our sound similarity metries, we find that self-supervised learning improves performance by 34.5%
supervised learning.
To summarize, this paper makes three contributions. First, we collect an audio-behavior dataset of 25k examples on five tasks that capture contact-rich, dynamic interactions. This dataset will be publicly accessible on our project website. Second, we show that self-supervised learning techniques can significantly improve the performance of behavior prediction from audio. These contact-rich dynamic behaviors are learned solely from audio inputs. Third, perform analysis various design choices such as the self-supervised objective used and the amount of data needed for training.
2 Related Work
Our framework builds on several important works in audio-based methods in robotics, self- supervised learning, and dynamic manipulation. In this section, will briefly describe works that are most relevant to ours.

---

Audio for Robotics: The use of audio information has been extensively explored in the context of multimodal learning with visual data [11, 13, 14, 10, 15]. Several of our design decisions like the use of convolutional networks are inspired by this line of work [11]. In the context of robotics, several works have used audio for better navigation [16, 17, 12, 18], where the central idea is that sound provides useful signal of the environment around the robot. Recently, the sound generated by quadrotor has been shown to provide signal for visual localization [19]. The use of audio for manip- ulation remains sparse [7, 9, 20, 8]. Prior work in this domain has looked at using sound to improve manipulation with granular material [7], connecting sound with robotic planar manipulation [9], and learning imitation-based policies for multimodal sensory inputs [20]. We draw several points of inspiration from these works including the use of contact microphones to record sound [7. 9] and training behavior models [20]. AURL differs from these works in two aspects. First, we focus on dynamic manipulation that generates sound through contact-rich interaction. Second, we show that such manipulation behaviors can be learned without any visual multimodal data.
Self-Supervised Learning on Sensory Data: Self-supervised representation learning has led to impressive results in computer vision [21, 22, 23, 24] and natural language processing [25, 26]. The goal of these methods is to extract low-dimensional representations that can improve down- stream learning tasks without the need for labeled data. This is done by first training a model on 'pretext' tasks with an unlabeled dataset (e.g. Internet images). These pretext tasks often include instance invariance augmentations such color jitters rotations [27, 28, 21, 22, 29]. The use of such self-supprvised learning has recently shown promise in visual robotics tasks such as door opening [30] and dexterous manipulation [31].
In the context of audio, several recent works have explored self-supervised representation learning for speech and music data [32, 33, 34, 35, 36, 37, 38, 39]. AURL takes inspiration from BYOL- A [32] to the BYOL [23] framework for learning auditory representations. However, unlike prior works that use music speech data train their representations, our data includes multi- channel contact sound that requires special consideration. For example, we found that the Mixup augmentation used in B YOL-A does not work well on our contact sound data.
Dynamic manipulation: Training dynamic manipulation behaviors, i.e. behaviors that require dynamic properties such as momentum, has received significant interest in the robotics commu- nity [40, 41, 42, 43] Several works in this domain learn policies that output parameters of prede- fined motion primitives [44, 45], which makes them amenable supervised learning approaches. Our work uses similar action parameterization for the dynamic tasks we consider. However, in contrast to many of these works [46, 45, 47], AURL can operate in domains where visual informa- tion does not contain sufficient temporal resolution or contact information to solve desired tasks.
3 Audio-Behavior Dataset Collection
Since there are no prior works that have released datasets to study the interaction of audio with dynamic behavior learning, we will need create our own datasets. We select tasks that allow for seff-supervised dataset collection while being dynamic and contact-rich (see Fig. for task visualization). For tasks that involve making contact with the environment, we use deformable tools to ensure safety. All of our data is collected on a UR10 robot, with behaviors generated through joint velocity control, and sound generated by dynamic contacts recorded by contact microphones.
3.1 Dynamic Tasks and Setup
Our dataset consists of five tasks: rattling a rattle, shaking a tambourine, swatting fly swatter, strik- ing a horizontal surface, and striking a vertical surface. For each task, we attach a different object to our UR10's end effector with 3D printed mounts. Dynamic motion for each task is generated by using motion primitives that control the robot's velocity and acceleration profile. After the execu- tion of the robot motion, distinctive sounds are produced either from the object itself (e.g. rattle) or through interaction with the environment (c.g. swatter). Audio generated from each interaction
3

---

is recorded for 4 seconds at 44.1kHz by contact microphones either attached to the robot or in the environment. To visualize behaviors and train vision-based models, we also record RGB visual data from a third-person camera.
For each task, we collect data for 1, 0,0 unique behaviors, with 5 examples of each behavior: This collection of multiple examples is done to account for the noise present in audio data even with the same behavior applied. This amounts to 5, 000 datapoints per task; for a total of 25, 000 datapoints in our dataset. Collecting each data point takes "10s (including time to safely reset the robot), for a total of "14 hours of data collection being run continuously on a robot for each task. To automate the collection then we sample these parameters from uniform distribution. The sampled primitives are used to generate trajectory that is then executed the robot operating in joint velocity control mode. The sampled parameterization of primitives and the self-resetting nature of the tasks allows data collection to proceed with minimal human supervision. For tasks that have parameters that control spatial positions of object interaction, we attach multiple microphones to better capture the spatial relation between sound and action. Details of each of our tasks are as follows:
diverse dataset, first we parameterize the robot's s action space
Rattling rattle: We attach a rattle to the robot's end-effector. The rattle can generate different behaviors through parameters that control the velocities, accelerations, and number of oscillations, to generate rich variations in sound. This task uses single contact microphone on the rattle to record audio. The primitive is defined by the elbow joint velocity, acceleration and number of oscillations.
Similar to the rattling task, we attach a tambourine to the end-effector. The tambourine can be moved with different velocities, accelerations, and number of oscillations. This task uses a single contact microphone placed on the tambourine attachment. Similar to the rattle, the primitive is defined by the elbow joint velocity, acceleration and number of oscillations
Shaking a tambourine:
S Swatting a fly swatter: We attach a flexible fly swatter to the end-effecctr. The robot then executes a swatting motion on a board placed in front of the robot on the table's top. This task uses two contact microphones on opposite ends of the board to record the audio. The primitive is defined by the velocity of the base joint, the shoulder joint and the acceleration. The acceleration and velocity of the base determines the location of where the flyswatter strikes the board and that of the shoulder joint determines the velocity with which the robot strikes the board.
Striking Horizontally: We attach soft ball to the end of a rope to the robot end-effector. The robot then executes a swinging/dragging motion to strike a plywood board placed horizontally in front of the robot on a closed box. We record the audio produced by this motion with two contact microphones placed on adjacent sides of the box. The action space is defined by the shoulder, elbow and wrist velocities, acceleration, number of time steps for which each joint moves. Varying the number of time steps for which each joint moves allows for more diverse behaviors such as dragging and striking to be generated.
Striking Vertically: We use a configuration similar to the striking horizontally task with a de- formable ball attached to the end-effector. Instead of horizontal plane, the robot generates swinging motion that allows the ball strike vertical plywood board. We record the audio pro- duced by this motion with two contact microphones placed adjacent sides of the vertical surface. The action space is defined by shoulder, elbow and wrist velocities and acceleration. We first apply an initial velocity to generate momentum for the rope. Then we apply a different set of velocities to generate the swinging motion which produces the audio on impact with the open box. Adding momentum allows us to generate more diverse dynamic behaviors.
4 AURL
4.1
Overview
Each task in our dataset can be represented as D = fox: a where on and as represent the audio information and the action information respectively for datapoint k. Given this training data,

---

Multi-chaned andio ingut
Spectrogram
Augmentations
Encoder
Representation
Action parametens
Prodeces
Audio Preprocesing
Self supervised Training
AationAction Finctuming
Figure 2: AURL framework: Before training, we preprocess the audio data for compatibility with CNNs. Then, we train an encoder using self-supervised training with BYOL. Finally, we finetune a linear model to predict actions that generate dynamic behaviors the robot
wish to learn policy f(xx) äg that can predict actions from input sound data. One straight- forward approach learn this is by using standard supervised learning. However, such a learning methodology often requires extensive real-world data that impractical collect for many of the dynamic tasks consider.
We present two-step approach to efficiently learn with a moderately sized dataset. For each task, we first use state-of-the-art self-supervised learning to train concise aural representations using only our observation data, i.e. {or This model can represented as λ사 = go (ox), where 2ạ is the low-dimensional representation obtained from the high dimensional audio observation og- Second, train supervised linear probe on top of the representation 2k: i.e. âR w/ (=x). We hence able leverage high-capacity neural networks in learning representations while preventing overfitting during supervised training by using low-capacity linear models. Our learning framework is illustrated in Fig. 2, with details of individual components in the following sections.
4.2 Self-Supervised Representation Learning
Conventionally, learning with audio has been primarily used in the speech and music do- mains [48, 49]. While several pretrained models leverage audio for speech recognition and music manipulation, there is little work connecting audio to dynamic robotic actions that generate sound. Since the audio from actions differs significantly from speech and music in terms of rhythm, tone, and amplitude, we find that using pretrained audio models does not generalize well to robotic tasks (see Section 5). Hence, to leverage the relatively modest size of training data compared to internet scale audio [50, 51J, we rely on self-supervised representation learning.
In this work, inspired by prior work in representation learning for robotics [30, 31], Bootstrap Your Own Latent (BYOL) [23] the self-supervised method to train encoder. The key idea in BYOL is to learn representations that are robust to augmentations in the input. BYOL does this using two networks, the online network and the target network, which are trained in parallel. The network's objective is to minimize the distance between the representations of different augmented versions of the same audio input. The target network updated as slow exponential moving average of the online network to ensure stationarity of the target representation. In our implementation, normalize the data and use random resize and crop augmentations on the mel spectrogram of the audio inputs.
4.3 Audio to Behavior Finetuning
After training the encoder with the self-supervised objective, we use a linear model Wy to transform the low-dimensional encodings 2ạ to the behavior parameters ag. This procedure mimics common
practice in computer vision probe self-supervised models [23, 22, 21J. The linear weights
optimized by gradient descent to minimize the squared loss Σn_ [w 2 l We provide additional training details in Appendix A.
5

---

Experiments
We note that this representation for similarity deviates from metrics music and speech recognition such as chromagrams{$6) or MFCCs[57]. This is because metries in music focus on the temporal nuances (pitch and tone) in sharp sounds created by our robots and tend to ignore the general shape of the audio waveform. In contrast, the DTW metric does not assume that the signals have the same of the We evaluate AURL a variety of dynamic manipulation experiments. These experiments
designed to answer the following questions:
Can AURL learn dynamic behaviors from audio?
Do AURL behaviors generate desired sounds?
How important is self-supervised learning in AURL?
What components and design choices AURL are essential to performance?
5.1 Baseline methods
To contextualize the results from AURL, we run the following methods on our data:
Here we select random action from the training set. This should serve as lower bound on performance.
Random:
Here each test point is matched to the closest train point, i.e. the train action that has the smallest L2 distance to the test action. This should serve as rough upper bound on performance.
Oracle:
Supervised: Here we randomly initialize a ResNet [52] encoder along with linear layer as the regression head, and train it from scratch using the MSE objective.
Supervised+Aug Here we follow the same training procedure as the supervised model, but we augment the input data with the same augmentations used in AURL
Pretrained audio: Here we extract auditory representations from a pretrained data2vec [53] model, followed by training a linear model to predict the action. This serves as an alternative to the self- supervised pretraining used in AURL.
Prefrained vision: Here we extract 5 equally spaced frames from the recorded video of the robot data. The frames are then passed through frozenImageNet [50] pretrained ResNet [52] encoder extract representations. We concatenate these representations and learn predict the action using a linear probe. We only use 5 frames since using a larger number of frames requires significantly larger computational resources compared to the audio models trained in AuRL.
5.2 Evaluation metrics
To quantify each method, we use the following metrics:
MSE action prediction: To quantify the performance of different models we will primarily use the mean squared error (MSE) between the predicted behavior parameters and the ground truth parameters on the test set.
DTW sound similarity: To evaluate the similarity between two audio signals, we first take the amplitude envelope of the raw audio signals to remove high-frequency noise. We then use Dynamic Time Warping (DTW) [54, 55] to align the two amplitude envelopes. The distance between the two envelopes after DTW will
raw distance metric. We then normalize this score, as (x ()/o where is the distance, μl, and is the mean and standard deviation of the raw distance values of sounds generated by the same action applied on the robot.
length are even aligned in time. Instead, it aligns the two signals in time by warping signals so that it best matches the other signal. This makes it suitable for tasks where the signals may have different lengths or may be shifted in time, both of which are expected when comparing two audio signals.

---

Table 1: MSE error (4) on the prediction of behavior parameters for baselines and AURL
Rattling
Shaking
Swatting
Striking (H)
Striking
(V)
Random
2.36
2.61
0.67
1.10
1.06
Oracle
0.07
0.07
0.06
0.31
0.32
Sup
0.25
0.28
0.09
0.59
0,45
Sup+Aug
2.37
2.19
0.27
0.76
0.58
Pre. Aud
3.10
4.19
0.25
0.65
0.53
Pre. Vid
2.30
3.05
0.05
0.52
0,48
AURL
0.12
0.22
0.02
0.53
0.39
AURL+Act
0.12
0.23
0.02
0.58
0.41
AURL+AA
0.14
0.26
0.01
0.54
0.38
AURL+MT
0.12
0.15
0.01
0.60
0.43
5.3 Can AURL learn dynamic, contact-rich behaviors from audio?
To quantitatively evaluate the behaviors learned, we compare the MSE between the action predicted by the learned model and the ground truth action used to generate the behavior on the robot. A summary of these results with the MSE metric is in Table We see that the self-supervised meth- ods outperform other competitive approaches. Particularly for the Striking Vertically and Swatting tasks, we find that our trained models can achieve a performance very close the Oracle method. Interestingly, find that supervised learning with augmented data performs quite poorly, which highlights the importance of using supersuup to extract information from augmentations.
5.4 Do the AURL behaviors generate desired sounds?
To gauge the dynamism of the produced behaviors and ensure fidelity to the input audio, we run our trained models on our URIO robot and measure the similarity of the generated sounds with the input desired sound (see Appendix B) using the normalized DTW metric introduced in 5.2. We summarize the results of this experiment in Table 2. We observe that AURL methods outperform baselines on 4/5 tasks. In the Shaking task, supervised training performs slightly better than the self supervised methods. In tasks such as Striking(H) and Rattling, we notice that AURL can outperform the Oracle method, which indicates that self supervision can be useful in extracting good represen- tations. This demonstrates the promise of self-supervision to interpolate and select actions that are not present in the training set.
5.5 How important is self-supervised learning in AURL?
Instead of using the self-supervised objective in AURL, one could also use pretrained representa- tional models that were trained on much larger, albeit out-of-distribution data [50, 58]. We compare against two such pretrained models. The first uses pretrained audio models from data2vec [58], and the second uses pretrained visual models learned from ImageNet data. As we see in Table both these pretrained models perform quite poorly. This is not surprising since the data used to pretrain the audio models consists of music and speech data, while our task requires understanding contact sounds. Similarly, image-basee based models due to their low temporal frequency cannot capture information relevant to predicting dynamic behavior.
5.6 What components and design choices in AURL are essential to performance?
We experiment with several design choices from our dataset collection setup to our learning framework.
Contact microphones: At the hardware level we found that using contact microphones instead of standard microphones allowed to record sounds with significantly lesser ambient noise (e.g. people speaking in the background). This reduces the need for post-processing to denoise the data.

---

Table 2: Normalized DTW distance (1) between the amplitude envelopes of the desired audio and produced audio.
Rattling
Shaking
Swatting
Striking (H)
Striking (V)
Random
4.95
8.33
7.12
6,05
3,36
Oracle
1.07
7.61
4.47
3.24
1.85
Sup
1.10
7.67
5.86
2.43
2.53
Sup+Aug
6.51
8.22
5.80
3.81
2.37
AURL
0.96
7.76
5.23
2.35
2.05
AURL+AA
0.97
7.68
4.98
2.18
2.13
Scaling with data size: We studied the effects of data scale with performance by running our method across variety of different slices of training data collected in AURL. We find that although increasing the amount of data helps, self-supervised pretraining leads to larger relative improvements in the low data regime. (See Appendix C for more details). Hence, like several large-scale machine learning works [26, 52]. it leads us to conjecture that while at the limit of data size plain supervision in the low and moderately sized data regimes which specialized self-
may be sufficient, supervision techniques might be crucial. Further, collecting a large dataset is time intensive. For better performance with limited data, self supervised learning may be beneficial.
Types of augmentation: We found that the standard audio augmentation of Mixup [32] performs quite poorly with our data. This is understandable since mixing different audio signals loses the temporal information needed to predict behaviors. Similar to recent works in reinforcement learn- ing [59] we find that using just a few augmentations (random resize and crop) works quite well for contact sound data. We note that commonly used image augmentation techniques like rotation and gaussian blur are not meaningful when applied to audio data (spectrograms).
Mechanisms of self-supervision: Instead of running standard BYOL, our control over the dataset generation process allows us to collect data that explicitly contains the invariances we require from auditory representations. For example, instead of specifying the invariance through data augmenta- tions, we can specify the invariance by using two audio signals generated by the same behavior. As denoted by AURL+Act in Table we see that although we obtain gains against supervised meth- ods it does not perform well as AuRL. However, we find that using this invariance along with augmentations, denoted AURL+AA, improves aural similarity performance on 3/5 tasks.
Multi-task pretraining: To evaluate if task specific pretraining is beneficial for AURL, we pretrain the AURL encoder using standard BYOL on all of the audio data collected across all tasks. As de- noted by AURL+MT in Table we see that AURL+MT performs at par with other AURL methods for 2/5 tasks and performs better in only 1/5 tasks. One reason for this is that the striking tasks show sparse sharp spikes in the waveform/ spectrogram while rattling, shaking and swatting show dense spikes in the spectrogram. Due to the large variation in the distributions of the spectrograms between tasks we posit that task-specific pretraining is ideal for our set of tasks.
6 Limitations and Discussion
In this paper we have presented AURL, framework to train self-supervised auditory representations for dynamic manipulation with contact. While we show improved results with audio inputs, there are currently two limitations of this work. First, for tasks such as Striking Horizontally and Shaking a Tambourine we find only mild gains over plain supervised learning on the MSE action prediction metric, while being significantly poorer than the Oracle method. Further research improving robotic self-supervision or leveraging large-scale audio data can significantly improve these models. Second, current framework focuses primarily on learning the interactions from labeled datasets, which limits our models to only capture behaviors present in our training set. Expanding AURL to reinforcement learning based dataset collection can allow for task-specific learning.

---

Acknowledgments
If paper is accepted, the final camera-ready version will (and probably should) include acknowl- edgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.
References
[1] N. Schaffert, T. B. Janzen, K. Mattes, and M. H. Thaut. A review the relationship between sound and movement in sports and rehabilitation. Frontiers in psychology, 10:244, 2019.
[2] A. O. Effenberg, U. Fehse, G. Schmitz, B. Krueger, and H. Mechling. Movement sonification: effects on motor learning beyond rhythmic adjustments. Frontiers neuroscience, 10:219, 2016.
[3] S. Ghai, I. Ghai, G. Schmitz, and A. O. Effenberg. Effect of rhythmic auditory cueing on parkinsonian gait: a systematic review and meta-analysis. Scientific reports, 8(1):1-19, 2018.
[4] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[5] I.. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. ICRA, 2016.
[6] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. ISER, 2016.
[7] Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. Proceedings of Machine Learning Research, 87, 2018.
[8] H. Liang, S. Li, X. Ma, N. Hendrich, T. Gerkmann, F. Sun, and J. Zhang. Making sense of audio vibration for liquid height estimation in robotic pouring. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5333-5339. IEEE, 2019.
[9] D. Gandhi, A. Gupta, and L. Pinto. Swoosh! rattle! thump'-actions that sound. arXiv preprint arXiv:2007.01851, 2020.
[10] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and A. Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16867-16876, 2021.
[11] A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adelson, and W. T. Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405-2413, 2016.
[12] C. Chen, S. Majumder, Z. Al-Halah, R. Gao, S. K. Ramakrishnan, and K. Grauman. Audio- visual waypoints for navigation. arXiv preprint 2XXi:2008.00962., 2020.
[13] R. Arandjelovic and A. Zisserman. Look, listen and learn. CoRR, abs/ 1005 705.08168, 2017.
[14] Y. Aytar, C. Vondrick, and A. Torralba. Soundnet: Learning sound representations from un- labeled video. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 892-900. Curran Associates, Inc., 2016.
[15] T. Afouras, A. Owens, J. Chung, and A. Zisserman. Self-supervised learning of audio-visual objects from video. In European Conference on Computer Vision, pages 208 2244.24. Springer, 2020.

---

[16] X. Wu, H. Gong, P. Chen, Z. Zhong, and Y. Xu. Surveillance robot utilizing video and audio information. Journal of Intelligent and Robotic Systems, 55(4):403-421, 2009. 2009.
[17] V. Dean, S. Tulsiani, and A. Gupta. See, hear, explore: Curiosity via audio-visual association. Advances in Neural Information Processing Systems, 33:14961-14972 2020.
[18] Z. Chen, X. Hu, and Owens. Structure from silence: Learning scene structure from ambient sound. arXiv preprint arXiv:211 2021.
[19] M. Nava, A. Paolillo, J. Guzzi, L. M. Gambardella, and A. Giusti, Learning visual localization of a quadrotor using its noise as self-supervisiin. IEEE Robotics and Automation Letters, 2022.
(2):2218
[20] M. Du, O. Y. Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. arXiv preprint arXiv:2205. 2022.
[21] Chen, S. Korblithh, K. Swersky, M. Norouzi, and G. Hinton. Big supervised models are strong semi-supervised learners. arXiv preprint (xXx:20066/0029 2020.
[22] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.042: 042997 2020.
[23] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self- supervised learning, arXiv preprint arXiv:2006.07733 2020.
[24] M. Caron, Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0995 2020.
[25] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.00 1810.04805 2018
[26] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[27] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[28] I. Misra, L. Zitnick, and M. Hebert. Shuffle and learn: Unsupervised learning using temporal order verification, 2016.
[29] Z. Wu, Y. Xiong, S. Yu, and D. Lin. Unsupervised feature learning via non non-paramtr instance- level discrimination, 2018.
[30] J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint 11205221 2021.
[31] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A baseed framework for efficient dexterous manipulation. arXiv preprint arXiv:2203. 13251, 2022.
[32] D. Niizumi. D. Takeuchi. Y. Ohishi, N. Harada. and K. Kashino. Byol for audio: Self- supervised learning for general- purpose audio representation. In 202/ International Joint Con- ference on Neural Networks (IJCNN) pages 1-8. IEEE, 2021.
[33] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self- supervised learning of speech representations. Advances in Neural Information Processing Systems, 05112449-24460 2020.
10

---

[34] D. Jiang, W. Li, M. Cao, W. Zou, and X. Li. Speech simelr: Combining contrastive and reconstruction objective for self-supervised speech representation learning. arXiv preprint arXiv:2010.1399) 2020.
[35] M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Roblek. Pre-training audio repre- sentations with self-supervision. IEEE Signal Processing Letters, 27:600-604 2020.
[36] Ma, R. Mira, S. Petridis, B. W. Schuller, and M. Pantic. Lira: Learning visual speech representations from audio through self-supervision. arXiv preprint arXiv:2106.0977,, 2021.
[37] M. Cartwright, J. Cramer, J. Salamon, and J. P. Bello. Trieycle: Audio representation learning from sensor network data using self-supervision. In 2019 IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 278-282. IEEE, 2019.
[38] A. Shukla, S. Petridis, and M. Pantic. Learning speech representations from raw audio by joint audiovisual self-vuuervision. arXiv preprint arXiv:2007.00 040000000414,, 2020.
[39] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong. J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint (xXiv:2003004290 2020.
[40] K. M. Lynch and M. T. Mason. Dynamic underactuated nonprehensile manipulation. In Pro- ceedings of IEEE/RSJ International Conference Intelligent Robots and Systems. IROS 196 volume 2, pages 889-896. IEEE, 1996.
[41] K. M. Lynch and M. Mason. Dynamic nonprehensile manipulation: Controllability, planning, and experiments. The International Journal of Robotics Research, 18(1):64 92, 1999.
[42] S. Kim, A. Shukla, and A. Billard. Catching objects in flight. IEEE Transactions on Robotics, 30(5): 1065, 2014.
[43] Ichnowski, Y. Avigal, Y. Liu, and K. Goldberg. Gomp-fit: Grasp-optimized motion plan- ning for fast inertial transport. In 2022 International Conference on Robotics and Automation (ICRA), pages 5255-5261. IEEE, 2022.
[44] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 3674)11307-139,, 2020.
[45] B. Evans, A. Thankaraj, and Pinto. Context is everything: Implicit identification for dynam- ies adaptation. arXiv preprint arXiv:2203.055: 2022.
[46] H. Zhang, J. Ichnowski, D. Seita, J. Wang, H. Huang, and K. Goldberg. Robots of the lost arc: Self-supervised learning to dynamically manipulate fixed-endpoint cables. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4560 4567. IEEE, 2021.
[47] H. Ha and S. Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pages 24-33. PMLR, 2022.
[48] A. Gulati, J. Qin, C.-C. Chiu, Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, al. Conformer: Convolution-augmentee transformer for speech recognition. arXiv preprint arXiv:2005.08100 2020.
[49] Z. Fu, G. Lu, K. M. Ting, and D. Zhang. A survey of audio-based music classification and annotation. IEEE transactions on multimedia, 13(2):303-313 2010.
[50] O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. In arXiv preprint urXiv:1409.0577 2014.

---

[51] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, IEEE, 2017.
[52] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[53] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-super learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
[54] Senin. Dynamic time warping algorithm review. Information and Computer Science De- partment University of Hawaii at Manoa Honolulu, USA, 855(1-23):40, 2008.
[55] H. Sakoe and Chiba. Dynamic programming algorithm optimization for spoken word recog- nition. IEEE transactions on acoustics, speech, and signal processing, 26(1):43-44 49, 1978.
[56] C. Harte and M. Sandler. Automatic chord identifcation using quantised chromagram. In Audio Engineering Society Convention /18. Audio Engineering Society, 2005.
[57] V. Tiwari. Mfce and its applications in speaker recognition. International journal on emerging technologies, 1(1):19-22, 2010.
[58] Huggingface:
Data2vec.
https://muggiigaace.coom Accessed: 2022-09-15.
[59] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved
autteeugeeeted reinforcement learning. arXiv preprint arXiv:2107.09645 2021.
[60] I. Daubechies. The wavelet transform, time-freyuency localization and signal analysis. IEEE transactions on information theory, 36(5):961-1005, 1990.
[61] K. Choi, G. Fazekas, M. Sandler, and K. Cho. comparison of audio signal preprocessing methods for deep neural networks on music tagging. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1870-1874. IEEE, 2018.
[62] M. Dörfler, Bammer, and T. Grill. Inside the spectrogram: Convolutional neural networks in audio processing. In 2017 international conference on sampling theory and applications (SampTA), pages 152-155. IEEE, 2017.
[63] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. Large batch optimization for deep learning: Training bert 76 minutes. arXiv preprint arXiv: 1904. 00962, 2019.
[64] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /41 /1.. 69800 2014.
12

---

A Training Details
Since we convolutional networks to fit our audio-action data, will need to first process the audio to have compatible dimensions. To ensure that all audio data have the same length of 4s, we clip longer sequences and pad shorter audio sequences to equal lengths. Next, we downsample the audio from 4k.zkHz to 1 1ikHz to reduce the dimensionality of the data. To make our audio data compatible with CNNs, we can either use plain STFT [60], mel, or log- mel [61] spectrogram. We choose the mel spectrogram representation over the other options based on initial experiments. To do this, STFT of window size 400 and hop length 160 is first applied on the downsampled audio, followed by the application of a mel filter [62] with 16 mel banks. To ensure compatibility with different number of microphones, we treat data from each microphone as separate channel. This representation allows us to treat ID audio observations as 2D images.
Audio preprocessing:
Self-supervised training: For of our experiments with self-supervised learning, we use ResNet-18 18 [52] encoder with the number of channels equal to the number of contact microphones used for the corresponding task. The length of the latent representation 2 for all tasks is fixed to 512. Following standard practice [52], we normalize the mel-spectrograms of the audio with channel-wise mean and variance. We train each encoder for 1000 epochs on the task's audio data. We use the LARS [63] optimizer with an initial learning rate of 0.2, weight decay of 1.5e-6, and batch size 1024.
Finetuning: For each of our tasks, split our dataset of 5000 datapoints into 4000 train audio- action pairs and 1000 unseen test audio-action pairs. We train the linear weights wy with batch size of 1024, the Adam [64] optimizer with learning rate of 10-A, and weight decay of 10-4.
We normalize our inputs with mean and standard deviation of the each channel. For the output of each our models, for all tasks except shaking and rattling, to predict the actions, we perform min max normalization of actions from the train dataset. While evaluating our model on the robot, we scale the predicted actions with the minimum and maximum calculated during training. For tasks that do not use normalized action space, clip the maximum and minimum of the action space. We do so to ensure that the robot operates within safety limits.
B Qualitative Results
Ratting
Swatting
Horixontal
Stiking Verical
Figure 3: Waveforms of audio generated by robot during the evaluation of different models. For tasks using multiple microphones, we visualize the waveforms from a single microphone. We see that the models learned with AURL better match desired audio compared to standard supervised approaches, particularly on the Striking Horizontal task.
13

---

C
Scaling effects of dataset size
Suticapii
Sepornsol
Sepervied
AdkI
dezpotor
Figure 4: Comparison of AURL and supervised learning on MSE error (L) when varying the number of unique actions used in training. We see that in the low data regime, self-supervised pretraining through AURL significantly improves upon vamilla supervised learning. However, there are dimin- ishing returns when the dataset size increases.
14


Reviews:
[{"review_id": "H2mDjxf3j6", "reviewer": "Reviewer_UsSe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper explores the concept of motor learning through the use of auditory inputs. The tasks examined are those typically underperformed when relying solely on visual data. This includes actions like rattling a rattle or shaking a tambourine, where the completion of contact-rich tasks is heavily reliant on sound. The authors collected a robust dataset comprising 5000 data points per task. The tasks evaluated include a range of activities including rattling a rattle, shaking a tambourine, swatting with a fly swatter, and performing horizontal and vertical strikes using a UR10 robot. The proposed model outputs actions given a desired sound as input. \n\nThe proposed model transforms the audio signal into a spectrogram, then a CNN is used to extract features. These features are trained using the BYOL target, before a linear model translates these features into corresponding actions.\nAn important finding in this study was the use of self-supervised learning using the BYOL method.", "summary_of_recommendation": "This paper introduces an novel approach to motor learning, where actions are learned from desired audio inputs. This novel setting could potentially contribute fresh insights to the field of multi-modal robot manipulation, which leads me to recommend acceptance. However, I must point out that the experiments presented within the paper do not fully substantiate the claims made by the authors. Additionally, the use of a self-supervised learning approach during feature extraction is not particularly groundbreaking. I would suggest the authors to enhance their experimental design and offer more compelling evidence to support their claims.\n\n-------------\nPost rebuttal: My questions has been addressed and I recommand weak accept", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The 'Oracle' method, as it is presented in the paper, appears to be a nearest-neighbor approach. As per my understanding, it operates by identifying the closest audio sample within the training data given a desired auditory target input, and then mimics the corresponding action from the training sample. If this is the case, the use of the term 'Oracle' seems to be misleading, as this approach doesn't take more inputs than other methods. From the experiment results, it seems that this conventional nearest-neighbor method is better than training neural networks. \n\n- The paper's motivation to use auditory-only inputs is unconvincing. The introduction of multi-modal inputs could potentially enhance task completion. Most manipulation tasks, particularly context-rich ones, greatly benefit from visual inputs, making them essential for learning actions.\n\n- Building upon the previous point, the paper claims that the tasks under investigation cannot be accomplished using solely visual inputs. To support this claim, the author should provide a more thorough evaluation of the visual baselines used for comparison. Several critical technical details remain undisclosed. For instance, what is the duration of the five frame inputs used in the vision baseline, and how does this compare to the audio inputs? Is the ImageNet pretrained ResNet features frozen during the process or are they fine-tuned?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is well-written and easy to comprehend\n- The task of learning context-rich tasks through auditory inputs is interesting. \n\nWeaknesses:\n- The ablation study provided within the paper lacks compelling evidence \n- I'm not fully convinced with the setting of using audio inputs without combining them with visual inputs. The motivation behind this approach needs to be clarified.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "BOa33Uy9pzl", "reviewer": "Reviewer_KaUb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduce AuRL, a self-supervised network for generating robot motion from a sound input.\nAuRL is composed of two parts: a self-supervision/encoder network that defines latent audio features; a fine tuning network that predicts the motion to generates a similar sound as the input.", "summary_of_recommendation": "Given the many technical and experimental flaws, this paper is not currently at the level of a CoRL paper.\nIt is highly recommended to check the soundness of all equations, improve the clarity of the paper and provide additional details about the experimental study.\n\nPost Rebuttal Update:\nThe reviewer appreciates the efforts and rigorous answers of the authors in the rebuttal process.  The reviewer regrets that the same rigorousness and efforts were not put in the first version of the paper (with obvious \"amateurish\" flaws). If the paper is accepted, it requires some substantial reworking from version 1. Despite the potential benefit for the robot audition community, because of the above  weaknesses, the reviewer updates his recommendation to weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) Please specify clearly the inputs and outputs of the proposed method.\n2) Please detail the method figure. The figure is too shallow.\n3) Line 161, $a_k$ is not a policy from the reviewer understanding. It's the output. The policy is $f$.\n4) Line 167, what are the purposes of the indexes $1$ and $n$?  The same were used in line 159 but never defined.\n5) Line 169, what is the meaning of $\\theta$\n6) Line 194, rigorously the squared loss function is wrong.  With the author's notation this would be equal to 0, because of the definition of $a_k$ line 169. Again, somewhere should be defined the ground truth and predicted value. This notation error principally comes from the fact that the input and outputs were not correctly and clearly specified.\n7) Please improve the clarity  and organization of the experimental section  as mentioned in the weaknesses above.\n8)The authors overlooked/did not mention clearly,  that the BYOL encoder is trained separately for each different task. Each task seems to have different number actions (i.e. output size). So a custom AuRL network is needed for each task. This a  clear practical limitation in the reviewer opinion.\n9) Please detail the intra-task data distribution ? What are the different sound collected from the same task ?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Globally this work lacks many details and rigorousness, which confuse the reader and drastically reduce the quality of the paper.\nStrengths:\n- Original and relevant research area.\n- Experimental validation of the approach\n\nWeaknesses:\n- Unclear methodology. In baseline supervised/unsupervised, It is crucial for the reader to understand from the start  the inputs and outputs of the proposed method. Yet, after reading the paper, the reader is unable to understand the output of the proposed method.  How is the robot controlled (discrete primitives? velocity control? sequence?).  This is a major flaw.\n- Many inaccurate/undefined terms in equations\n- Experimental evaluation unclear: The MSE is computed for which behavior parameters?  What is the unit? How is the L2 distance computed? What is a test point?\n- The organization of the experimental results is confusing. It start from Table 1 to Table 2 then back to Table 1...", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9AsgKs6K5-K", "reviewer": "Reviewer_N4ak", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper pushes robot learning using audio input. Given a sound, a robot is tasked with creating the sound with behaviours. They use self supervised learning technique to learn latent representations of audio, then a second network to learn behaviour given a latent representation. Mean square error over produced sound is used as an accuracy measure. The paper benchmarks against a few known baselines and does ablations over supervised learning and a couple other design choices.", "summary_of_recommendation": "I like this paper, it's an intersecting problem set up, an unusual and infrequently explored data modality. the hypotheses generated are sound and the experiments make a reasonable shot at answering them. Qualitative results on actual audio samples will greatly improve this paper. I would accept this paper, so more roboticists can think about working with audio data. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Why is the random baseline better than pretrained audio baseline? That's weird?\n2. The website doesn't have qualitative results. MSE is a good metric but it is not a palpable metric, some examples of sounds produced by robot after learning could be useful to understand where things stand. \n3. What are the main failure modes of this approach? A little bit on this would be useful.\n4. Would the self supervision result generalize to generic data constrained regimes?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like this paper, its very humorously written and is a surprisingly innovative approach to robot learning. That is refreshing. I am not an expert in audio tech but the methods used look reasonable to me. \n1. For the table 1, the abbreviation for methods in the lower blocks, it is not apparent to me what they stand for. Would be nice to explain below.\n2. The result that self supervised learning is better than supervised pretraining in data constrained regimes is an important observation.\n3. Limitations and discussions sections could be improved\n4. An interesting question is, can we do audio conditioned task execution using similar methods or speech-to-actions?\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "-ipkDx-uwW", "reviewer": "Reviewer_KfWt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors collect a audio dataset and use self-supervised learning to improve performance of behavior prediction from audio.", "summary_of_recommendation": "I think the framework is interesting and the dataset is quite novel since it is a visual audio dataset for robotic manipulation. Hence, I recommend accept the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The audio dataset for contact-rich manipulation is very interesting and novel. My only thought is that it would be more interesting to see if this dataset can be used to improve the performance of other more complex manipulation tasks. \n\nThere are previous work that leverage audio to learn contact-rich dynamic manipulation (See, Hear, and Feel by Li and Play it by ear by Du). But their limitations are that they are collecting their own task-specific audio dataset and train their model from scratch. I think some of the behaviors in your paper are representative in contact-rich manipulation tasks (like the collision sound, it would be better to have some scratch sound as well). It might have bigger impact to the community if this dataset can be generalized to other tasks and other people can used it for pretraining.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n*The authors presents a novel audio dataset for robotic manipulation.\n*Sufficient experiments are conducted on five different tasks to demonstrate the increase in performance.\n\nWeaknesses:\n*Limitation in the use of this dataset. It would be more interesting to see if pretraining on this dataset will have performance increase in some more complex robotic manipulation tasks(eg. peg in hole task, training from scratch vs. pretraining on this dataset).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission initially received mixed reviews.  After discussion, the reviewers generally leaned toward acceptance based on the technical innovations and the experiments.  The AC agrees.  The authors are encouraged to further revise the paper based on the reviews in the camera-ready version.
Decision: Accept (Poster)