ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Authors: Zeyi Liu
Cheng Chi
Eric Cousineau
Naveen Kuppuswamy
Benjamin Burchfiel
Shuran Song
Venue: CoRL 2024 (2024)
Keywords: Robot Manipulation
Imitation Learning
Audio
TL;DR: We introduce ManiWAV: a data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations.

Abstract: Audio signals provide rich information for the robot interaction and object properties through contact. These information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments, by learning from diverse in-the-wild human demonstrations. All data, code, and policy will be public.

ManiWAV: Learning Robot Manipulation from In-the-Wile
Data
Zeyi Liu'
Cheng Chi
Eric Cousineau"
Naveen Kuppuswamy
Benjamin Burchfiel Shuran Song!2
Stanford University
Columbia University TToyota Research Institute
htps://manivav-iihhubie
Abstract: Audio signals provide rich information for the robot interaction and object prop- erties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone ambiguous or incomplete. However, the usage of audio data robot manipulation has been constrained to teleoperated demonstrations collected by either attaching microphone to the robot object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV. an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize unseen in in-the-wild environments by learning from diverse in the-wild human demonstrations.
Keywords: Robot Manipulation, Imitation Learning, Audio
furry
rough
d) Object States Contact sound reveals rich information. More specifically, (a) contact events, such as eraser touching the whiteboard; (b) different contact modes, such as spatula poking on the side of the bagel versus sliding below the bagel; (c) surface material, such as the furry ('loop') and rough ("hook") side of velcro tapes; (d) object states, such whether the cup contains objects not.
Contact Events
b) Contact Modes
c) Surface Materials
Fig 1:
1 Introduction
Selecting and executing good contact the core of robot manipulation. However, most vision-based robotic systems nowadays are limited in their capability to sense and utilize contact information. In this work, we propose a robotic system that learns contact through common yet under-explored modality audio. Our first insight is that audio signals provide rich contact information. During a manipulation task, audio feedback can reveal several key information about the interaction and object properties, including:
Contact events and modes: From wiping surface to flipping object with spatula, audio feedback captures salient and distinct signals that can be used for detecting contact events and characterizing contact modes (Fig. 1 a, b).
8th Conference Robot Learning (CoRL 2024), Munich, Germany.

---

1 Contact
2 Standard Recording
Microphone
S Wide-FoV
Interface (mic port)
Fisheye Lens
Robot
i Natural
Haptic
Feedback
High-Friction Surface
(a) Human Demonstration
(b) Audio Observation
(c) Robot Deployment
Fig 2: Ear-in-hand gripper for in-the-wild data collection. (a) The handheld design naturally provides haptic feedback to the demonstrator during contact-rich tasks (c.g., wiping), which is otherwise hard to obtain via teleoperation. Contact microphone captures high-frequency audio feedback that is recorded simultaneously with images. High friction tape is applied on top to augment the signals. (b) shows the domain gap between training and deployment data. (c) shows policy learned from nn-thewwld data directly deployed on the robot.
Surface materials: Audio signals can be used to characterize the surface material through contact with the object. In contrast, either image sensors or vision-based tactile sensors require high spatial resolution to capture the subtle texture difference (e.g.. the 'hook' and 'loop' side of velcro tapes) (Fig. 1 c).
Object states and properties: Without the need for direct contact, audio signals can provide complementary information about the object state and physical properties beyond visual observation (Fig. d).
Second, audio data is scalable for data collection and policy learning. This because acoustic sensors (i.e., contact microphones) are cheap, robust, and readily available to purchase. Audio signals also have standardized coding formats that can be easily integrated into existing video recording and storage pipelines (e.g., MP4 files). These nice properties make possible to collect audio in the wild with low-cost data collection devices, such as hand-held gripper, without the need for a robot. On the other hand, alternative ways to sense contact, such as tactile sensors, are relatively more expensive, fragile, and require expert knowledge to use.
Given the richness and scalability of audio data, we propose versatile robot learning system, ManiWAV, that leverages audio feedback for contact-rich robot manipulation tasks. Building upon the portable hand-held gripper finger embed a piezoelectric contact microphone
data collection device UMI [1], we redesign that senses audio vibrations through contact with solid objects. The audio signals can be easily streamed to the GoPro camera through a mic port and stored synchronously with vision data in MP4 files. With this intuitive design, one can demonstrate a wide range of manipulation tasks with synchronous vision and audio feedback at a low time and maintenance cost.
To learn from the collected demonstrations, one key challenge is to bridge the audio domain gap between in-the-wild data and actual robot deployment due to test-time noises (Fig. 2 b). To achieve this goal, we propose a data augmentation strategy that encourages learning of task-relevant audio representation. In addition, we propose an end-to-end sensorimotor learning network to encode and fuse the vision and audio data, with a diffusion policy [2] head for action prediction.
We demonstrate the capability of our proposed system on four contact-rich manipulation tasks: wipe shape from whiteboard, flip bagel with spatula, pour objects from cup, and strap wires with velero tape. We also show that our system can generalize to unseen in-the-wild environments by leveraging in-the-wild data collected from diverse environments.
2 Related Work
Tactile Sensing for Contact-rich Manipulation. As functional equivalent to the human touch modality, tactile sensing has long been studied to provide feedback for the robot's physical interactions [3]. Ranging from six-axis force/torque sensors [4, 5] to camera-basse tactile sensors [6. 7, 8, 9, 10, 11] and tactile skins [12, 13, 14, 15], tactile sensors take various forms and provide information of contact forces [16), object geometry [17, 18], etc. Many recent works have incorporated tactile feedback into robot learning pipelines for contact-rich manipulation tasks by learning better visuo- tactile representations [19, 20, 21, 22, 23, 24]. However, most tactile sensors are limited in their reproducibility given the cost and requirement for domain

---

knowledge to use. In this work, we find that acoustic sensors such as piezoelectric contact microphones can provide alternative tactile feedback at a much lower cost and higher availability.
Acoustic Sensing in Controlled Environments. Sound is an important information carrier of the physical environment. Acoustic sensing can be categorized into active and passive. Active acoustic sensing is done by emitting vibration waveform with speaker, which is then received by a microphone. Prior works have embedded active acoustic sensors on objects [25], robot arms [26. 27], parallel gripper [28] or soft finger [29] to sense object material, shape, and contact interactions. Passive acoustic sensing captures sounds generated from interactions, and prior works have shown that including audio as input to end-to-end robot learning algorithms improves the performance of manipulation tasks [30, 31, 32, 33). However, the collection of audio data in prior works requires controlled environment, where the data is collected through teleoperation, with sensors attached either the robot the object. To address this limitation, propose 'ear-in-hand" gripper design with passive acoustic sensing collect human demonstrations without the need for robot, making it promising to collect in-the-wild audio-visual data at lower cost of time.
Policy Learning from Multisensory Data. Multisensory observations (c.g.. vision, audio, tactile) allow robots to better perceive the physical environment and guide action planning [19, 34, 35, 36]. However, many prior works require pre-defined state abstractions [37, 38] learn the control policy and are thus limited in their ability generalize to new tasks. Recently, end-to-end models have been proposed take in multisensory inputs and output robot actions through behavior cloning approach [33, 31, 32, 30J. We extend upon prior works and propose an audio data augmentation strategy to bridge the domain gap between in the- data and robot data during deployment, together with an end-to-end policy learning network to effectively learn task-relevant audio-visual representations from multimodal human demonstrations.
3
Method
We propose a data collection and policy learning framework to learn contact-rich manipulation tasks from vision and audio. On the data collection front, our goal to easily collect in-the-wild demonstrations with clean and salient contact signals. To achieve this, we extend the UMI [1] data collection device by embedding a piezoelectric contact microphone in one gripper finger to stream audio data synchronously with the vision data through the GoPro camera mic port.
On the algorithm front, one key challenge is to bridge the audio domain gap between the collected demonstrations and feedback received during robot deployment, as illustrated in Fig. (b). Another challenge learn a robust and task-relevant audio-visual representation that can effectively guide the downstream policy. To address these challenges, we propose data augmentation strategy bridge the audio domain gap and transformer-based model that learns from human demonstrations with vision and audio feedback.
3.1 Ear-in-Hand Hardware Design
Our data collection device built on the Universal Manipulation Interface (UMI) [1]. UMI a portable and low-cost hand-held gripper designed to collect human demonstrations in the wild. The collected data can be used to train visuomotor policy that is directly deployable robot.
We redesign the 3D-printed parallel jaw gripper on the device to embed a piezoelectric contact microphone under high-friction grip tape wrapped around the finger. The microphone is connected to the 3.5mm external mic port on the GoPro camera media mod, Fig. (a) shows the hand-held gripper design. Audio recorded 48000 Hz and stored with 60Hz image data synchronously as MP4 files. During robot deployment, the same parallel jaw gripper with embedded contact microphone is mounted Fig. 2 (c). The images and audio are streamed in real-time through an Elgato HD60 X external capture card into a Ubuntu 22.04.3 desktop.
URS robot arm, shown in
3.2 Policy Design
We propose an end-to-end closed-loop sensorimotor learning model that takes in RGB images and audio, and outputs 10-DoF robot actions (including end effector positions, end effector orientation represented in 6D [39], and ID gripper openness).
3

---

Audio Data Augmentation. One key challenge is that the audio signals received during real sal-ttime robot deployment are very different from the data collected by the hand-held gripper, resulting in a large domain gap between the training and test scenarios, as illustrated in Fig. 2 (b). This mostly because of 1) nonlinear robot motor noises during deployment and 2) out-of-distribution sounds generated by the robot interaction. (e.g.. accidentally colliding with an object).
To address the domain gap, the key is to augment the training data with noises and guide the model to focus on the invariant task-relevant signals and
End effecte pouts
CLIP WII
ignore unpredictable noises, In particular, we randomly sample audio background noises from ESC-50 [40]. The sounds normalized to the same scale as the collected sound in the training dataset. We also record 10 samples of robot motor noises under randomly sampled trajectories with the same contact microphone location as deployment time. The background noises and robot noises are overlayed to the original audio signal, each with a probability of 0.5. In our experiments, we show that this simple yet effective approach yields better policy performance by enforcing the inductive bias of the model task-relevant audio signals.
Image Encoder
Trundfemme Encoddr
Piab-aye
Linear Projection
conditiin
AST Enooder
Diffusion Policy
Fig 3: Network Architecture.
Vision Encoder. We use a CLIP-pretrained ViT- B/16 model [41] to encode the RGB images. The images are resized into 224x224 resolution with random crop and color jitter augmentation. Images sampled at 20 Hz. and take images in the past 2 timesteps. Each image encoded separately using the classification token feature.
Vicion only
Ours (Vision Audio)
Audio Encoder. We use the audio spectrogram transformer (AST) [42] to encode the audio input. AST, similar to VIT model, leverages the attention mechanism learn better audio representation from spectrogram patches. The intuition behind using a transformer encoder instead of a CNN-based en- coder as seen in prior works [31. 33, 30] is that the 'shift invariance' that CNN leverages is less suitable to audio spectrograms, as shifts in either the time and frequency domain can significantly change the audio information. In our experiment, show that training the transformer encoder from scratch outperforms both pre-trained and from-scratch CNN models.
Fig 4: Attention Visualization. Interestingly, we find that policy co- co-trained with audio attends more to the task-relevant regions (shape of drawing or free space inside the pan). In contrast, the vision-only policy often overfits to background structures as shortcut to estimate contact (e.g., the edge of the whitehoard, table, and room structures).
The audio signal (from the last 2-3 seconds, depending on the task) is first resampled from 48kHz to 16kHz, which helps filter high-frequenee noises and increase the frequency resolution task-relevant signals on the spectrogram. The waveform is then converted to log mel spectrogram using FFT size and window length of 400, hop length of 160, and 64 mel filterbanks. The log-mel spectrograms are linearly normalized to range [-1,1]. We use the classification token feature extracted from the last hidden layer of the AST encoder.
Sensory Fusion. We fuse the vision and audio features using a transformer encoder in a similar fashion as Liet al. [33] to leverage the attention mechanism to weigh the features adaptively at different stages of the task (c.g., vision is important for moving to the target object, whereas audio is important upon contact). We concatenate the output features and downsample the dimension to 768 with a linear projection layer. Finally, we concatenate the end effector poses (20 Hz) from the past 2 timesteps to the audio-visual feature.
Policy Learning. To model the multimodality intrinsic to human demonstrations, we choose to use diffusion model with UNet encoders as proposed by Chi et al. [2] as the policy head, conditioned on the observation representation mentioned above in each denoising step. The entire model (Fig. 3), including

---

diff. configs
audio perturbation
heights
T4: In wn-the-wild Test
Scenarios
Training
height
Ir
Vision only: poking
All: lose contact
In-lab: drifts away
Faiture Cases
On the left, we show the in-lab test scenarios. We train the policy with in-lab demonstrations collected in the same environment as inference time. We study three types of test-time variations: including different robot and object configurations (TI), audio perturbation when we play different types of noises in the environment (T2), and different table heights than the one in training data (T3). On the right, show the unseen environments for the nn-the-wild generalization test. Bottom: Typical failure cases and task success rate. The details (e.g., failure cases) for each rollout can be found in the appendix.
Fig 5: Flipping Evaluation. Up:
the above-mentioned encoders, is end-to-end trained using the noise prediction MSE loss on future robot trajectories of 16 steps.
Audio Lateney Matching, During data collection, the vision and audio data are synchronized when recording through GoPro. During deployment, calibrated the audio latency to be 0.23; details can be found in the appendix. We adopt an approach similar to Chi et al. [I] to compensate for this delay.
4
Evaluation
We study four contact-rich manipulation tasks to show the different advantages of learning from audio feedback, such as detecting contact events and modes (flipping and wiping), or sensing object state (pouring) and surface material (taping). In each task, we test the policy under different scenarios and compare with alternative approaches to validate the robustness and generalizability of our approach.
4.1 Flipping Task
The robot is tasked to flip a bagel in a pan from facing down to facing upward using a spatula. To perform this task successfully, the robot needs to sense and switch between different contact modes precisely insert the spatula between the bagel and pan, maintain contact while sliding, start to tilt up spatula when the bagel is in contact with the edge of the pan.
Data collection: We collect two types of demonstrations for this task: 283 in-lab demonstrations and an additional 274 in-the-wild demonstrations collected in different environments using different pans.
Test Scenarios: We run 20 rollouts for each policy. To ensure a fair comparison, we use the same set of robot and object configurations for evaluations between different methods. We achieve the same object configuration by overlaying their position with respect captured image in the camera view. The test configurations can be grouped into four categories.
Tl: Variations task configuration: different initial robot and object configurations (14/ 20).
T2: Audio perturbation by playing different types of noises in the background (2/ 20).
T3: Generalization unseen table height (4 20).
T4: Generalization to two unseen in-the-wild environments: black stovetop and a white countertop, the
latter is more challenging due to unstructured background and lack of similar training data.
Comparisons: In this task, we focus on comparing our results with several ablations of the network design:
Vision only: the original diffusion policy conditioned on image observations.
MLP policy: using an MLP with three hidden layers (following al. [33]) instead action diffusion.
The model takes the observation representation and outputs the future action trajectory.
5

---

T1:diff. diff. configs
T2: audio perturbation
T3: diff. shapes
T3: diff. erasers
T3: diff. table heights
Training height
Scenarios
No noise aug: press
Vision only: float
MLP tusion: incomplete
Failure Cases
6.25
06/3 Videe
HLP Place
1000 meniing
Fig 6: Wiping Evaluation, Up: Different test scenarios. Bottom: Typical failure cases and task success rate. [Vision only] policy often fails to maintain proper contact (e.g.. either press too hard into the broad or float). [MLP fusion] policy often fails fully wipe out the drawing and terminates early.
ResNet: uses a 8 encoder to encode the audio log-mel spectrograms, with an additional CoordCony layer [43] following Li et al. [33].
AVID: Following the approach by Mcjia al. [30], a 9-layer CNN audio encoder pre-trained on AudioSet [44] using Audio- Visual Instance Discrimination (AVID) [45].
In-the-wild: For in- tn-the-wild evaluation (T4), we compare our model trained with in-the-wild demonstrations
(blue bar Fig. bottom right chart), only in-lab demonstrations (red bar in Fig. bottom right chart), and [Vision only] baseline trained on n--lhe-wild data (yellow bar in Fig. 5 bottom right chart). Details the in-the-wild dataset be found in the appendix.
The quantitative result and typical failure cases are visualized in Fig. Our key findings are: Action diffusion yields better behavior compared to MLP, yet the [MLP policy] that takes in audio data still significantly outperforms [Vision only] diffusion policy. Action diffusion better captures the multimodality in human demonstrations; for example, one might approach the bagel from different directions in different demonstrations. 2) Transformer audio encoder outperforms a CNN-based encoder. We find that training a transformer encoder from scratch yields good performance compared to using CNN-based encoder, as shown the [ResNet] and [AVID] baseline. This is likely because the self-attention mechanism the transformer allows the model focus more on the frequency regions of task-relevant signals in the spectrogram. 3) In in-the-wild data enables generalization unseen in-the-will wild environments. As shown in Fig. 5, policy trained on tithe-wild data significantly outperforms in-lab data on two unseen environments, as the scene diversity in in wild data allows the policy to generalize better to new environments.
Findings:
4.2 Wiping Task
In this task, the robot is tasked to wipe shape (e.g., heart, square) drawn on whiteboard. The robot can start in any initial configuration above the whiteboard and grasp an eraser in parallel to the board. The main challenge of the task is that the robot needs to exert a reasonable amount of contact force on the whiteboard while moving the craser along the shape. We collect 119 demonstrations total for this task.
Comparisons: In addition to the vision only baseline, evaluate the following altematives for processing and learning from audio data:
MI.P fusion: uses an MLP with hidden layers to fuse the vision and audio features instead of a transformer encoder. This approach was used by Du et al. [31].
Noise masking: without noise augmentation but instead mask out the audio frequency helow 500 Hz, which is the UR5 control frequency.
No noise aug: without augmenting the audio data with noises during training.
Test Scenarios: We run 20 rollouts for each policy. addition to the TI / 20) and T2 (4 20) test cases as described above, also test generalization to unseen table heights, erasers, and drawing shape T3 (11 20). A detailed breakdown of the test scenarios can be found in the appendix.

---

Task Definition
T3: #dices
T3: different items
Init Typical Failure Cases
Grasp Shake
Pour
Final: Place Cup
Vision: stuck stucklseorr
before pour
1s: stuck after pour
10s: place w.o. pour
All: missitaried
Fig 7: Pouring Evaluation. In the first row, show the task definition and generalization scenarios for an unseen number of dice and items. The second row shows the typical failure case for each baseline method and the overall task success rate. A breakdown of the substep success rate can be found in the appendix.
Findings: The quantitative result and typical failure cases are visualized in Fig. 6. We find that: 1) Contact audio improves robustness and generalizability. Using only images from the wrist-mount camera, it is hard to infer whether the eraser is contacting the board or not, whereas incorporating the contact audio improves the overall success rate from 40% 85%. The [Vision only] policy fails to generalize to unseen table heights and unseen erasers. To understand the behaviors of the vision only policy and our policy better, we visualize the attention map of the vision encoder in Fig. 4 and find that the model trained with audio aftends better to task-relevant features. 2) Noise augmentation is an effective strategy to bridge the audio domain gap and increase the system's is robustness to out-of out-of-disse of-diitrrbution sounds. Without noise augmentation, the robot is less robust to noises during test time and does not generalize well to unseen table heights, unseen erasers, and unseen shapes. Another alternative we study is simply masking out the robot noise regions. Surprisingly, this approach yields slightly better results than [No noise aug] by removing the domain gap caused by robot motor noises. However, this alternative does not successfully address other noises during the robot execution, as mentioned in Section 3.2. Lastly, show 3) the advantage of using transformer fuse the vision and audio features compared using MLP. A typical failure in [MLP fusion] is that the robot fails wipe in the area of the shape and stops when the shape is not completely wiped off.
4.3 Pouring Task
The robot is tasked to pick up the white cup and pour dice out the pink cup if the white cup not empty. When finished pouring, the robot needs to place the empty cup down designated location. The challenge of the task is that the robot cannot observe whether there are dice in the cup or not given the camera viewpoint both before and after the pouring action. We collect 145 deronnstrttios for this task, with a 'shaking' action to generate vibrations that can be captured by the contact microphone if there are dice inside the cup.
Comparisons: In addition the vision only baseline, we study how the length of the audio input affects the policy performance with ablations: using Is history audio and using 10s history audio.
We run 12 rollouts for each policy. In addition to the T1 and T2 scenarios, we also tested generalization scenarios (T3) to an unseen number of dice (e.g.. no dice or >> 6 dice) and unseen objects (e.g.. screws or beans).
Test Scenarios:
Findings: The quantitative result and typical failure cases are visualized in Fig. 7. Our key findings in the experiments are: 1) Combining with information-seekiin action, audio can provide critical state information beyond visual observations. As illustrated in the figure, the vision only policy fails to execute the pour action as cannot infer whether there are dice in the cup or not, whereas the policy trained with audio feedback can leverage vibrations to infer the state information and generalize to objects with similar sounds (e.g., screws). 2) Policy performance is sensitive to audio history length. As shown the result, [Is audio] yields significantly lower performance since the shortened audio window does not contain sufficient information to guide robot actions. [10s audio] shows that even though longer audio window contains sufficient information, increases the complexity of the learning process and requires a higher capacity model.
7

---

Task Definition
BRED
Init
Touch every tape
Grosp the 'hook' tapa
Apply
Final
Typical Failure Cases
Vision wrong
All: touch
Altmmis
Fig 8: Taping Evaluation. In the first row, we show the task definition. The second row shows the typical failure case for each method and the overall task success rate. A breakdown of the substep success rate can be found the appendix.
4.4 Taping Task
The rohot tasked to choose the 'hook' tape from several velcro tapes (either 'hook' or 'loop') and strap wires by attaching the 'hook' tape a 'loop' tape undernathh the wires. We collect 193 demonsttrations in total, with a 'sliding' primitive where use the tip of the gripper finger to slide along the tape. We run 10 rollouts in total, each time the robot is presented 2-4 velero tapes in random order with at least one 'hook' tape.
Comparisons: In addition to vision only, we compare the following baselines:
Env Mic: Instead of using a contact microphone, we mount Rode VideoMic GO II directional microphone the GoPro camera for both data collection and deployment collect the audio signals.
Noise Reduction: Instead of applying training time noise augmentation, evaluated altemative method that uses test-time noise reduction algorithm. The algorithm estimates noise threshold for each frequency and applies smoothed mask on the spectrogram [46]. More details can be found in the appendix.
Findings. 1) Contact microphone is sufficiently sensitive to different surface materials. As shown in Fig. 8, the [Vision only] policy makes random decisions and yields a 20% success rate. Similarly, the system that uses environment microphone to collect audio data achieves similar results the [Vision only] method, fails pick up the subtle differences between the surface material. In contrast, by leveraging the contact microphone, our method is able reliably guide the robot to pick up the correct tape. 2) Training-tine time noise augmentation is more effective than test-time noise reduction. This is because the noise cancellation algorithm causes the signal to be deprecated, resulting in a domain gap between training and testing. On the other hand, our noise augmentation method preserves the frequency distribution of the original signals. More visualization can be found in the appendix.
4.5 I Limitations and Future Directions
On the data front, even though the contact microphone can pick up wide range of audio signals and is robust against environment noises in the background by design, it may not be useful in scenarios where the interaction does not generate salient signals (e.g., for deformable objects such as cloth or quasi-static tasks), and can easily become imperceptible due to robot motor noises during deployment. On the policy front, the current method does not leverage the fact that audio signals are received a higher frequency than images and can be used leam more reactive behaviors. Future work can consider hierarchical network architecture [47] that infers higher frequency actions from audio inputs.
5
Conclusion
Audio signals reveal rich information about the robot interaction and object properties, which can ease the leamning of contact-rich robot manipulation policies. We present MamiWAY an in-the-wild mudio-visuse data collection and policy learning framework. We design an 'ear-in-hand" gripper to collect human demonstrations that be directly used train robot policies through behavior cloning, By learning an effective audio-visual representation as condition to the diffusion policy, our method outperforms several alternative approaches on four contact-rich manipulation tasks and generalizes to unseen tn-the-wild environments.

---

Acknowledgments
The authors would like to thank Yifan Hou and Zhenjia Xu for their help with discussions and setup of real-world experiments, Karan Singh for assistance with audio visualization and data collection. In addition, we would like to thank all REALab members: Huy Ha, Mandi Zhao, Mengda Xu, Xiaomeng Xu, Chuer Pan, Austin Patel, Yihuai Gao, Haochen Shi, Dominik Bauer, Samir Gadre for fruitful technical discussions and emotional support. The authors would also like to thank Xiaoran "Van' Fan especially for his help with task brainstorming and audio expertise. This work was supported in part by the Toyota Research Institute, NSF Award #21 #2143601, #2132519, and Sloan Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the sponsors.
References
[1] C. C.i, Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipu- lation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint wwXiv:2402.10322 2024.
[2] Chi, Feng, Y. Du, Xu, E. Cousineau, Burchfiel, and Song. Diffiusion policy: Visuomotor policy learning via action diffusion. arXiv preprint urXiv:2303.041:: 2023.
[3] Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, and H. J. Ritter. A review of tactile information: Perception and action through touch. IEEE Transactions on Robotics, 36(6):1619-1644 2020.
[4] U. Kim, D.H. Lee, B. Kim, D.-Y. Scok, and H. R. Choi. A novel six-axis force/torque sensor for robotic applications. IEEE/ASME Transactions mechatronics,
2016.
[5] M. Cao, Laws, and R. Baena. Six-axis force/torque sensors for robotics applications: A review.
IEEE Sensors Journal, 21(24):2233827255) 2021.
[6] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[7] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation. In 2079 2nd IEEE International Conference on Soft Robotics (RoboSoft), pages 597-604. IEEE, 2019.
[8] Shimonomura. Tactile image sensors employing camera: review. Sensors, 19(18):933,, 2019.
[9] N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasey, T. Ikeda, and R. Tedrake. Soft-bubble grippers for robust and perceptive manipulation. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9917-9924. IEEE, 2020.
[10] G. Zhang, Y. Du, H. Yu, and M. Y. Wang. Deltact: A vision-based tactile sensor using dense color pattern. IEEE Robotics and Automation Letters, 7(4): 10785, 2022.
[11] S. Zhang, Chen, Gao, W. Wan, Shan, H. Xue, Sun, Y. Yang, and B. Fang. Hardware technology of vision-based tactile sensor: A review. IEEE Sensors Journal, 22(22)21410-21427, 2022.
[12] Y. Luo, Y. Li, P. Sharma, W. Shou, K. Wu, M. Foshey, B. Li, T. Palacios, A. Torralba, and W. Matusik. Learning human-cn environment interactions using conformal tactile textiles. Nature Electronics, 4(3): 193-201, 2021.
[13] B. Xu, L. Zhong, G. Zhang, X. Liang, D. Virtue, R. Madan, and T. Bhattacharjee. Cushsense: Soft, stretchable, and comfortable tactile-sensing skin for physical human-robot interaction. arXiv preprint arXiv:2405.031 2024.
[14] J. Egli, B. Forrai, T. Buchner, Su, X. Chen, and R. K. Katzschmann. Sensorized soft skin for dexterous robotic hands. arXiv preprint 4404v:2444.1000 19448, 2024.

---

[15] S. Zhang, Y. Yang, F. Sun, L. Bao, J. Shan, Y. Gao, and B. Fang. A compact visuo-tactile robotic skin for micron-level tactile perception. IEEE Sensors Journal, 2024.
[16] J. Xu, Y. Jia, D. Yang, Meng, Thu, Z. Guo, S. Song, and M. Ciocarlie. Tactile-based object retrieval from granular media, arXiv preprint arXiv:2402.04556, 2024.
[17] J. Xu, S. Song, and M. Ciocarlie. Tandem: Learning joint exploration and decision making with tactile sensors. IEEE Robotics and Automation Letters, 7(4): 2022.
[18] J. Xu, H. Lin, S. Song, and M. Ciocarlie. Tandem3d: Active tactile exploration for 3d object recognition. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10401-10407. IEEE, 2023.
[19] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L.. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 1943-8950 IEEE, 2019.
[20] J. Kerr, H. Huang, A. Wilcox, R. Hoque, J. Ichnowski, R. Calandra, and K. Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. arXiv preprint (0XXii:2299/34442 2022.
[21] Y. Chen, M. Van der Merwe, A. Sipos, and N. Fazeli. Visuo-tactile transformers for manipulation. In 6th Annual Conference on Robot Learning, 2022.
[22] Y. Wi, Zeng, P. Florence, and N. Fazeli. Virdo++: Real-world, visuo-tactibe dynamics and perception of deformable objects. arXiv preprint arXiv:2210.03701, 2022.
[23] I. Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: sele-supe pre-traiine training of tactile representations with robotic play. arXiv preprint arXiv:2303. 12076, 2023.
[24] Y. Liu, X. Xu, W. Chen, H. Yuan, H. Wang, Xu, R. Chen, and Yi. Enhancing generalizable 6d pose tracking of in-hand object with tactile sensing. IEEE Robotics and Automation Letters, 2023.
[25] M. Ono, B. Shizuki, and J. Tanaka. Sensing touch force using active acoustic sensing. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, pages 355-358, 2015.
[26] X. Fan, D. Lee, Y. Chen, C. Prepscius, V. Isler, L. Jackel, H. Seung, and D. Lee. Acoustic collision detection and localization for robot manipulators. In 2020 JEEE/S/ International Conference on Intelligent Robots and Systems (IROS), pages 1529-9536 IEEE, 2020.
[27] X. Fan, R. Simmons- Edler, D. Lee, L. Jackel, R. Howard, and D. Lee. Aurasense: Robot collision avoidance by full surface proximity detection. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1763-1770. IEEE, 2021.
[28] S. Lu and H. Culbertson. Active acoustic sensing for robot manipulation. In 2023 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 3161-3168, 2023. 1oi: 109/0055000 1023.1034
[29] G. Zöller, V. Wall, and O. Brock. Active acoustic contact sensing for soft pneumatic actuators. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7966-7972. IEEE, 2020.
[30] J. Mejia, V. Dean, T. Hellebrekers, and A. Gupta. Hearing touch: Audio-visual pretraining for contact-rich manipulation. arXiv preprint ArXiv:2405. 28550. 2024.
[31] M. Du, O. Y. Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. arXiv preprint urXiv:2205. 14850, 2022.
[32] A. Thankaraj and L. Pinto. That sounds right: Auditory self-supervision for dynamic robot manipulation. arXiv preprint ar arXiv:2110001116, 2022.
10

---

[33] H. Li, Y. Zhang, J. Zhu, S. Wang, M. A. Lee. H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. arXiv preprint arXiv:22/2.03888, 2022.
[34] R. Gao, Z. Si, Y.-Y. Chang, S. Clarke, J. Bohg. L.. Fei-Fei, W. Yuan, and J. Wu. Objectfolder 2.0: A multisensorry object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference computer vision and pattem recognition, pages 11598-10608 2022.
[35] R. Gao, Y. Dou, H. Li, T. Agarwal, Bohg, Y. Li, L. Fei-Fei, and J. Wu. The objectfolder benchmark: Multisensory learning with neural and real objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 17276-17286 2023.
[36] Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction. In Conference on Robot Learning, pages 1468-3484. PMLR, 2023.
[37] V. Chu, R. A. Gutierrez, S. Cherooa,, and A. L. Thomaz. Real-time multisensory affordance-basee control for adaptive object manipulation. In 2019 International Conference on Robotics and Automation (ICRA), pages 7783-7790, 2019. doi: 10. 1100911..200011 17999,0
[38] See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion. Science Robotics, 4(26) 1233, 2019.
[39] Y. Zhou, C. Barnes, Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pages 5745-5753, 2019.
[40] J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015-1018. ACM Press. ISBN 978-1-4503-3455944 doi:10. 041i0001144/77333328806333 URL
[41] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, Heigold, Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
[42] Y. Gong, Y.A. Chung, and J. Glass.
Ast: Audio spectrogram transformer. arXiv preprint
arXiv22104.0177, 2021.
[43] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018.
[44] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780, 2017. 1 1AL11///SSSSSP217.792222
[45] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pages 12475-12486, 2021.
[46] T. Sainburg. timssinn/nnossereduee:: v1.0, June 2019. URL htt..//p///b...ogg//55//////ooo
[47] S. Saxena, M. Sharma, and O. Kroemer. Mrest: Multi-resoluuie sensing for real-time control with vision-language models. arXiv preprint arXiv:2401. 14502, 2024.
11

---

Appendix
A Method Details
A.1 Audio Latency Calibration
Similar to using clapperboard in film production, tap on the contact microphone and match the time between the frame when the finger is observed to be in contact with the microphone and the corresponding audio signal captured by the contact microphone, A visual illustration is shown Fig. The audio received about 0.06s after the image received. As a result, the total audio latency is 0.17 + 0.06 0.23s, where 0.17s is the calibrated image latency following the approach in [1].
3.4s
Fig 9
A.2 Model Details
Image Augmentation. Fach image is randomly cropped with a 95% ratio and then resized to its original resolution in the replay buffer. make the learned model more robust to different lighting conditions test time, we apply ColorJitter augmentation with brightness 0.3, contrast 0.4, saturation 0.5, and hue 0.08.
Spectrogram Parameters. There several parameters that control the time and frequency resolution of the resulting spectrogram image when converting the audio waveform to spectrogram. The window length is the length of the fixed intervals in which STFT divides the signal: it controls the time and frequency resolution of the spectrogram. Hop length is the length of the non-intersecting portion of the window lengths. The smaller the hop length is, the more times a particular audio segment will present STFT, and the more clongated the time-axis of the resulting spectrogram will For sample rate of 16kHz, we use the default window length, 400, recommended by torchaudio. It recommended that the hop length is around 1/2 of the window length; we choose 160 to slightly augment the signal pattern along the time-axis since contact signals are usually sparser in time compared to nature sounds or speech.
Transformer Encoder: We use one transformer encoder layer with heads to fuse the vision and audio features. We set the feedforward dimension 2048 and the dropout ratio to 0.0.
End-to-End Training Details. For each task, the entire model is end-to-end trained on 2 NVIDIA GeForce RTX 3090 GPUs for 60 epochs, with batch size of 64. We use the AdamW optimizer with Ir=le-4, betas=[0.95, 0,999], eps=1.0e-8, weight. Biight.decay=1.0edd and apply EMA (Exponential Moving Average) on the weights.
B Evaluation Details
B.1 In-the-Wite Data Collection
We collect 274 in the wild data for the bagel flipping task in total, including 52 in a conference room, 37 and 44 in two kitchens, 46 in an office, 76 on lounge tables, and 19 in a cafe, using 4 different pans. Examples of the environments are shown Fig. 10.
B.2
Result Details
B.2.1
Pouring Task
A breakdown of the success rate for each substep in the pouring task is shown in Tab. 'Grasp' is successful if the robot grasps the white cup stably in its end effector, 'Pour" is successful if the robot pours all objects in the white cup to the pink cup on the table, 'Place' is successful if the robot places the white cup on the table after pouring. By using audio feedback to infer the object state (whether there are objects in the cup not), our
Grasp
Pour
Place
OURS
90
100
90
Vision only
100
0
8.3
Is audio
91.7
30
0
10s audio
100
60
16.7
Table 1: Success Rate Breakdown.
12

---

Fig 10: Example Scenes in the In-the- Will Dataset.
(a)
Empty Cup
Cup with Dice
Cup with Screws
Fig 11: We visualize the audio spectrogram in the pouring task when the robot 'shakes' the cup to sense whether there are objects inside the cup. (a) shows the audio feedback of an empty cup, (b) and (c) shows the audio feedback for a cup with dice and with screws inside, respectively.
method is able to reliably guide the policy to pour objects and place the cup, whereas the baselines either never execute the pour action or the place action, resulting in low substep success rate. We also find that the policy can generalize to unseen objects such as screws. In Fig. 11, we visualize the audio spectrogram when the robot 'shakes' the cup. We can observe that the spectrogram for an empty cup is distinctive from a non-empty cup, and a cup with screws generates a similar audio pattern as a cup with dice upon shaking. We hypothesize that the audio features for the cup with screws and the cup with dice are also close in the audio feature space, leading similar policy behavior. Videos of the policy rollouts can be found on the project website.
B.2.2 Taping Task
breakdown of the success rate for each sub- step in the taping task is shown in Tab. 2. 'Touch' is successful if the robot slides along the tape while maintaining contact, 'Sense' is successful if the robot chooses the correct tape, 'Pick' successful if the robot successfully grasps the tape. 'Place" successful if the robot successfully places the tape on top of the wires. By leveraging audio feedback to infer the object surface material (whether the tape is 'hook" 'loop'), our method is able reliably guide the policy to choose the correct tape, whereas the baselines make random decisions, as shown in the 'Sense" step success rate. Videos of the policy rollouts can be found on the project website.
Touch
Sense
Pick
Place
OURS
90
90
80
80
Vision only
80
30
90
80
Env Mic
80
30
90
40
Noise Reduction
80
40
90
90
Table 2: Success Rate Breakdown.
Noise Reduction Algorithm. We use the non-stationary noise reduction method introduced in [46]. The algorithm computes spectrogram from the audio waveform, and then applies an IIR filter forward and backward on each frequency channel obtain time-smoothed version of the spectrogram. A mask is then computed based on the spectrogram by estimating noise threshold for each frequency band of the signal/noise. Finally, a smoothed, inverted version of the mask is applied to the original spectrogram to cancel
13

---

(8)
Training signal
(b) Real time signal
(c) Signal after noise reduction
Fig 12: Spectrogram Visualization.
noise. our experiments, we apply this algorithm directly to the real-time audio signals before feeding to the model for inference. In Fig. 12, we show spectrogram visualization of the real-time signal (b) and the signal after reduction (c). Even though the noise reduction seems to successfully remove most of the robot noises and some other background noises, it does not preserve the original signal well and still results in domain gap as compared to the training signal (a).
B.2.3
Wiping Task
We listed out each test scenario in the wiping task evaluation and success (1) failure (0) for each method.
Test Scenario
OURS
Vision only
MLP fusion
Noise masking
No noise aug
0
T1 (square)
0 (incomplete)
(press)
Tl (heart)
0 (incomplete)
(press)
2
(circle)
O (incomplete)
3
T3 (star)
(press)
0 (press)
4
(+1 inch)
(float)
0 (incomplete)
(incomplete)
5
(+5 inch)
(press)
0 (incomplete)
(press)
(press)
6
T3 (changing height)
0 (float)
0 (float)
(press)
7
T3 (changing height)
(press)
0 (incomplete)
(press)
(press)
8
(white noise)
(press)
1
9
(white noise)
10
T2 (construction noise)
(press)
0 (press)
11
(music)
(float) (float)
(incomplete)
(incomplete)
12
T1 (board orientation)
0 (float)
0 (float)
0 (float)
(incomplete)
13
T1 (board position)
1
(incomplete)
14
T3 (unseon eraser)
I
0 (float)
(press)
(press)
0 (press)
15
T3 (unseen eraser)
I
(incomplete)
(press)
16
T3 (-1 inch)
0 (float)
0 (float)
17
T3 (-1 inch)
0 (float)
0 (incomplete)
0 (press)
0 (press)
18
T3 (-2 inch)
(flott
0 (float)
0 (float
19
T3 (-2 inch)
1
0 (float)
0 (incomplete)
0 (press)
I
Success rate
85%
40%
35%
50%
35%
Table 3: Wiping Test Scenario Breakdown.
"Float' means the robot keeps floating above the hoard without contacting the board before it wipes off the shape. 'Press" means the robot exerts too much force downward and causes the gripper to bend against the board. 'Incomplete' means the robot fails to follow the shape, either stops wiping early or wipes in the wrong location.
As we can see from Tab. 3, 'Float' and 'Press" are most common in the [Vision only] baseline, especially when the table height is different than training, likely due the fact that it's insufficient infer contact from the top-down view
camera image alone (as shown in Fig. 13).
14

---

The most common failure case in [MLP fusion] is 'incomplete,, where the robot stops wiping and releases the eraser before the shape is completely wiped off. We hypothesize that this because simply fusing vision and audio features with MLP layers loses information that's crucial for inferring the stage of the task.
Without noise augmentation, the policy exhibits various unex- pected behaviors including 'press', 'incomplete', and 'float', because of the big domain gap between training and testing. The policy achieves better performance by simply masking out the robot noise frequency range in the [Noise masking] baseline, however, it still fails half of the time and most of the failure cases are 'pressing" too hard against the board.
Fig 13: Camera view in the wiping task.
B.2.4 Flipping Task
We listed out each test scenario in the bagel flipping task evaluation and success (1) failure (0) for each method.
Test Scenario
OURS
Vision only
MLP policy
ResNet
AVID
0
Tl
O (lose)
0 (poke)
T1
1
0 (lose)
0 (displace)
0 (lose)
2
T1
1
0 (poke)
0 (displace)
0 (displace)
3
T1
1
0 (lose)
0 (poke)
T1
1
0 (lose)
T1
O (lose)
0 (lose)
0 (displace)
6
T1
0 (lose)
7
T1
0 (poke)
0 (displace)
0 (poke)
1 O (poke)
8
T1
1
1
0 (displace)
9
T1
1
1
0 (lose)
10
T1
1
1
0 (displace)
T1
1
(poke)
0 (poke)
0 (displace)
12
T1
1
(poke)
13
T1
1
O (lose)
0 (lose)
0 (poke)
14
T2 (clap)
1
0 (poke)
O (displace)
1.
T2 (construction noise)
1
0 (poke)
0 (stuck)
16
T3 (unseen height)
1
0 (displace)
17
T3 (unseen height)
0 (lose)
(poke)
(lose)
0 (displace)
0 (lose)
18
T3 (unseen height)
1
0 (lose)
(poke)
(displace)
19
T3 (unseen height)
1
(displace)
(lose)
O (displace)
(displace)
Success rate
90%
25%
60%
35%
55%
Table 4: Flipping Test Scenario Breakdown.
'Poke' means that the robot pokes the spatula on the side of the bagel instead of inserting the spatula between the pan and the bottom of the bagel. 'Lose' means the robot loses contact with the bagel before it is flipped. As result, the bagel falls back to its original side. 'Displace' means that the spatula is displaced in the robot end effector as compared to its initial pose, as a result of the robot keeps moving down instead of switching to slide the spatula along the bottom of the pan.
For all T1 scenarios, we randomize the robot's initial pose and object positions (e.g., bagel and pan). We can observe that the [Vision only] policy does not generalize well across initial configurations and most failure cases are either poking on the side of the bagel (since it's hard to infer from the image alone if the spatula is contacting the bottom of the pan), or losing contact with the bagel carly before it can be flipped.
15

---

Using [ResNet] and [AVID] audio encoder results in the spatula to 'displace' most of the times, likely because the model not sensitive enough to the sound feedback of spatula touching the bottom of the pan, and as a result the robot keeps moving downward and causes the spatula to displace.
16


Reviews:
[{"review_id": "ZcwgWVA26p", "reviewer": "Reviewer_cdf4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Problem Studied by Authors: The authors in this work explore the feasibility of adding audio as a sensing modality to learn manipulation skills. This incremental work extends the prior work [34] cited by the authors in references. It is interesting to see how audio can help understand object properties (e.g., furry, rough, etc) and thus provide rich contact information. On a very high level, the authors have integrated audio and demonstrated how it can improve manipulation capabilities in addition to other modalities. Proposed Solution: Building on prior work, the authors introduce audio sensing to learn richer representations of manipulated objects. Their main contribution is leveraging audio to aid in learning a diffusion policy for executing contact-rich manipulation tasks.", "summary_of_recommendation": "Overall, this work is compelling and highlights the significant role of audio in enhancing the learning of contact-rich manipulation policies. Building upon UMI, it introduces an innovative approach for collecting human demonstration data in real-world environments. However, further clarifications are needed to address some gaps in the understanding of the paper.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Can the authors provide more insights into the design choices? Why use CLiP-pretained ViT? Specifically, when fusion is happening on audio data and ViT? Why not use something similar to Sung-Bin, Kim, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. \"Sound to visual scene generation by audio-to-visual latent alignment.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6430-6440. 2023. In this work, the authors have shown how one can align audio embedding with visual embeddings to generate visual scenes from audio. It is understandable that CLIP-trained ViT might give sufficient performance, but some insights might be useful for the reader\n2. Can the authors provide some evidence through preliminary experiments or prior work that the performance would be at par or comparable if wrench data was used? \n3. Would the intensity of background noise affect the performance?\n4. Have the authors given a thought to adding something like a GelSight sensor to the fingers? Specifically, are you placing the sensors where the black-colored high friction pads are present? This increases cost but might give better performance (This is more of a clarification question)\n5. Can the authors provide how sensitive the policy is to the number of demonstrations? Can using audio improve the data requirement for training diffusion policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LgDNe7WL4N", "reviewer": "Reviewer_AFoB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an 'ear-in-hand\" demonstration device for teaching robot manipulation skills. As they show, audio can be used as a cheap and scalable alternative for rich tactile information. In contrast to prior work, demonstrations can be carried out without the presence of the robot or any augmentation to the environment. This is made possible through their policy learning method which utilises data augmentation to focus the policy attention on task-relevant portions of the audio.", "summary_of_recommendation": "While this work has high potential impact in the field the novelty of the method is quite weak.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- What kind of background noise was present in T2? A more detailed study on the policy's robustness to different noise would interesting. For example, people talking, music, TV, etc... What happens if the noise is too loud and audio becomes unreliable?\n- Can you provide more details on what the difference is between in-lab vs in-the-wild data/demonstrations are?\n- In the wiping/ pouring tasks, it's not clear to me why the policy is able to generalise to unseen shapes/ objects. Can you provide more discussion on this?\n- Regarding the pouring task, \"the vision only policy fails to execute the pour action as it cannot infer whether there’s dice in the cup or not\". Can you provide more details on why this occurs? I would imagine that even a vision based policy should eventually learn the correct pouring behaviour. \n- Is the sensor fusion transformer pre-trained?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doIbGagZTE", "reviewer": "Reviewer_7TLA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a variation of UMI system with an embedded contact microphone. They use this handheld system to collect human demonstrations in-the-wild and present an end-to-end policy across four tasks (flipping, wiping, pouring, and taping). Within each experimental rollout, they define variations and ablations to compare the overall effect of their design choices such as data augmentation methods, noise filtering, or model architectures.", "summary_of_recommendation": "Despite similarities to previous work, I support a weak accept as the updated model architecture and data augmentation appear to have consistently improved results, which will be of interest to the community.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The portable system presented in the first paper contribution bears a strong resemblance to the UMI interface, as acknowledged by the authors through citation. The soft finger structure, tape, GoPro, and markers appear to be identical. Beyond the addition of a contact microphone, are there any other significant differences that make this a primary contribution of the work?\n\nWhy did the authors choose to resample from 48kHz to 16kHz? Can you also clarify if this means you filtering out the higher-frequency data or downsampled to ease the computational cost?\n\nFor those working with tactile audio, I believe they would benefit from additional reasoning for the selection of window/hop length and other preprocessing decisions near line 146.\n\nFor each of the four tasks, there are about 100-120 demonstrations. How was this number decided? Was their any motivation behind why some tasks required additional demonstrations? I would be very interested to see an ablation across number of demonstrations required. While your device makes the data collection process efficient, I believe it is still valuable to understand the data efficiency of this approach. \n\nSimilarly, why did the authors collect 200 demonstrations for the taping task? I am guessing 100 with the original contact microphone and 100 with the Rode microphone, but I did not see that distinction in the text. \n\nI found the summary of findings at the end of each section quite helpful and well-written. However, the Figures 5-8 were difficult to follow. Specially, Figure 5 bar charts are too small to read easily, and the overlaid text is hard to follow. Consider editing the caption to provide more context and moving some content to the appendix to emphasize your main findings.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-Rebuttal Guidance**

Recommendation: ACCEPT. All the authors unanimously suggested a weak accept. The only concerns the reviewers had was on novelty (minor improvement over prior work on UMI), however this was mitigated by both strong results demonstrating the importance of the changes and the author clarifications in the rebuttal. This will be a interesting paper for the CoRL community.

**Pre-Rebuttal Summary**

The paper proposes a new hardware (a UMI gripper augmented with a audio device) and approach to jointly learn from audio-visual signals.

Strengths:
- Paper is easy to follow, well written
- Excellent results demonstrated the usefulness of audio data
- Excellent real robot evaluation

Weaknesses:
- Over claimed contribution of the hardware, since the proposed hardware is a extension of UMI - authors please clarify this.
Decision: Accept