ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

Authors: zeyi  liu
cheng  chi
eric  cousineau
naveen  kuppuswamy
benjamin  burchfiel
shuran  song
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 20:22:57

Abstract: Audio signals provide rich information for the robot interaction and object properties through contact. These information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments, by learning from diverse in-the-wild human demonstrations. All data, code, and policy will be public.

Main paper: 
 introduction: 
 Selecting and executing good contact is at the core of robot manipulation. However, most vision-based robotic systems nowadays are limited in their capability to sense and utilize contact information. In this work, we propose a robotic system that learns contact through a common yet under-explored modalityaudio. Our first insight is that audio signals provide rich contact information. During a manipulation task, audio feedback can reveal several key information about the interaction and object properties, including:• Contact events and modes: From wiping on a surface to flipping an object with spatula, audio feedback captures salient and distinct signals that can be used for detecting contact events and characterizing contact modes (Fig. 1 a, b ).8th Conference on Robot Learning (CoRL 2024), Munich, Germany. • Surface materials: Audio signals can be used to characterize the surface material through contact with the object. In contrast, either image sensors or vision-based tactile sensors require high spatial resolution to capture the subtle texture difference (e.g., the 'hook' and 'loop' side of velcro tapes) (Fig. 1 c ). • Object states and properties: Without the need for direct contact, audio signals can provide complementary information about the object state and physical properties beyond visual observation (Fig. 1 d ).Second, audio data is scalable for data collection and policy learning. This is because acoustic sensors (i.e., contact microphones) are cheap, robust, and readily available to purchase. Audio signals also have standardized coding formats that can be easily integrated into existing video recording and storage pipelines (e.g., MP4 files). These nice properties make it possible to collect audio in the wild with low-cost data collection devices, such as a hand-held gripper, without the need for a robot. On the other hand, alternative ways to sense contact, such as tactile sensors, are relatively more expensive, fragile, and require expert knowledge to use.Given the richness and scalability of audio data, we propose a versatile robot learning system, ManiWAV, that leverages audio feedback for contact-rich robot manipulation tasks. Building upon the portable hand-held data collection device UMI [1] , we redesign one gripper finger to embed a piezoelectric contact microphone that senses audio vibrations through contact with solid objects. The audio signals can be easily streamed to the GoPro camera through a mic port and stored synchronously with vision data in MP4 files. With this intuitive design, one can demonstrate a wide range of manipulation tasks with synchronous vision and audio feedback at a low time and maintenance cost.To learn from the collected demonstrations, one key challenge is to bridge the audio domain gap between in-the-wild data and actual robot deployment due to test-time noises (Fig. 2 b ). To achieve this goal, we propose a data augmentation strategy that encourages learning of task-relevant audio representation. In addition, we propose an end-to-end sensorimotor learning network to encode and fuse the vision and audio data, with a diffusion policy [2] head for action prediction.We demonstrate the capability of our proposed system on four contact-rich manipulation tasks: wipe shape from whiteboard, flip bagel with spatula, pour objects from cup, and strap wires with velcro tape. We also show that our system can generalize to unseen in-the-wild environments by leveraging in-the-wild data collected from diverse environments. 
related work: 
 Tactile Sensing for Contact-rich Manipulation. As a functional equivalent to the human touch modality, tactile sensing has long been studied to provide feedback for the robot's physical interactions [3] . Ranging from six-axis force/torque sensors [4, 5] to camera-based tactile sensors [6, 7, 8, 9, 10, 11] and tactile skins [12, 13, 14, 15] , tactile sensors take various forms and provide information of contact forces [16] , object geometry [17, 18] , etc. Many recent works have incorporated tactile feedback into robot learning pipelines for contact-rich manipulation tasks by learning better visuo-tactile representations [19, 20, 21, 22, 23, 24] . However, most tactile sensors are limited in their reproducibility given the cost and requirement for domain knowledge to use. In this work, we find that acoustic sensors such as piezoelectric contact microphones can provide alternative tactile feedback at a much lower cost and higher availability.Acoustic Sensing in Controlled Environments. Sound is an important information carrier of the physical environment. Acoustic sensing can be categorized into active and passive. Active acoustic sensing is done by emitting a vibration waveform with a speaker, which is then received by a microphone. Prior works have embedded active acoustic sensors on objects [25] , robot arms [26, 27] , parallel gripper [28] or soft finger [29] to sense object material, shape, and contact interactions. Passive acoustic sensing captures sounds generated from interactions, and prior works have shown that including audio as input to end-to-end robot learning algorithms improves the performance of manipulation tasks [30, 31, 32, 33] . However, the collection of audio data in prior works requires a controlled environment, where the data is collected through teleoperation, with sensors attached to either the robot or the object. To address this limitation, we propose an 'ear-in-hand' gripper design with passive acoustic sensing to collect human demonstrations without the need for a robot, making it promising to collect in-the-wild audio-visual data at a lower cost of time.Policy Learning from Multisensory Data. Multisensory observations (e.g., vision, audio, tactile) allow robots to better perceive the physical environment and guide action planning [19, 34, 35, 36] . However, many prior works require pre-defined state abstractions [37, 38] to learn the control policy and are thus limited in their ability to generalize to new tasks. Recently, end-to-end models have been proposed to take in multisensory inputs and output robot actions through a behavior cloning approach [33, 31, 32, 30] . We extend upon prior works and propose an audio data augmentation strategy to bridge the domain gap between in-the-wild data and robot data during deployment, together with an end-to-end policy learning network to effectively learn task-relevant audio-visual representations from multimodal human demonstrations. 
method: 
 We propose a data collection and policy learning framework to learn contact-rich manipulation tasks from vision and audio. On the data collection front, our goal is to easily collect in-the-wild demonstrations with clean and salient contact signals. To achieve this, we extend the UMI [1] data collection device by embedding a piezoelectric contact microphone in one gripper finger to stream audio data synchronously with the vision data through the GoPro camera mic port.On the algorithm front, one key challenge is to bridge the audio domain gap between the collected demonstrations and feedback received during robot deployment, as illustrated in Fig. 2 (b ). Another challenge is to learn a robust and task-relevant audio-visual representation that can effectively guide the downstream policy. To address these challenges, we propose a data augmentation strategy to bridge the audio domain gap and a transformer-based model that learns from human demonstrations with vision and audio feedback.Our data collection device is built on the Universal Manipulation Interface (UMI) [1] . UMI is a portable and low-cost hand-held gripper designed to collect human demonstrations in the wild. The collected data can be used to train a visuomotor policy that is directly deployable on a robot.We redesign the 3D-printed parallel jaw gripper on the device to embed a piezoelectric contact microphone under high-friction grip tape wrapped around the finger. The microphone is connected to the 3.5mm external mic port on the GoPro camera media mod. Fig. 2 (a) shows the hand-held gripper design. Audio is recorded at 48000 Hz and stored with 60Hz image data synchronously as MP4 files. During robot deployment, the same parallel jaw gripper with embedded contact microphone is mounted on a UR5 robot arm, shown in Fig. 2 (c ). The images and audio are streamed in real-time through an Elgato HD60 X external capture card into a Ubuntu 22.04.3 desktop.We propose an end-to-end closed-loop sensorimotor learning model that takes in RGB images and audio, and outputs 10-DoF robot actions (including end effector positions, end effector orientation represented in 6D [39] , and 1D gripper openness).Audio Data Augmentation. One key challenge is that the audio signals received during real-time robot deployment are very different from the data collected by the hand-held gripper, resulting in a large domain gap between the training and test scenarios, as illustrated in Fig. 2 (b ). This is mostly because of 1) nonlinear robot motor noises during deployment and 2) out-of-distribution sounds generated by the robot interaction. (e.g., accidentally colliding with an object). To address the domain gap, the key is to augment the training data with noises and guide the model to focus on the invariant task-relevant signals and ignore unpredictable noises. In particular, we randomly sample audio as background noises from ESC-50 [40] . The sounds are normalized to the same scale as the collected sound in the training dataset. We also record 10 samples of robot motor noises under randomly sampled trajectories with the same contact microphone location as deployment time. The background noises and robot noises are overlayed to the original audio signal, each with a probability of 0.5. In our experiments, we show that this simple yet effective approach yields better policy performance by enforcing the inductive bias of the model on task-relevant audio signals. Interestingly, we find that a policy co-trained with audio attends more to the task-relevant regions (shape of drawing or free space inside the pan). In contrast, the vision-only policy often overfits to background structures as a shortcut to estimate contact (e.g., the edge of the whiteboard, table, and room structures).Vision Encoder. We use a CLIP-pretrained ViT-B/16 model [41] to encode the RGB images. The images are resized into 224x224 resolution with random crop and color jitter augmentation. Images are sampled at 20 Hz and we take images in the past 2 timesteps. Each image is encoded separately using the classification token feature.Audio Encoder. We use the audio spectrogram transformer (AST) [42] to encode the audio input. AST, similar to a ViT model, leverages the attention mechanism to learn better audio representation from spectrogram patches. The intuition behind using a transformer encoder instead of a CNN-based encoder as seen in prior works [31, 33, 30] is that the 'shift invariance' that CNN leverages is less suitable to audio spectrograms, as shifts in either the time and frequency domain can significantly change the audio information. In our experiment, we show that training the transformer encoder from scratch outperforms both pre-trained and from-scratch CNN models.The audio signal (from the last 2-3 seconds, depending on the task) is first resampled from 48kHz to 16kHz, which helps filter high-frequency noises and increase the frequency resolution of task-relevant signals on the spectrogram. The waveform is then converted to a log mel spectrogram using FFT size and window length of 400, hop length of 160, and 64 mel filterbanks. The log-mel spectrograms are linearly normalized to range [-1,1]. We use the classification token feature extracted from the last hidden layer of the AST encoder.Sensory Fusion. We fuse the vision and audio features using a transformer encoder in a similar fashion as Li et al. [33] to leverage the attention mechanism to weigh the features adaptively at different stages of the task (e.g., vision is important for moving to the target object, whereas audio is important upon contact). We concatenate the output features and downsample the dimension to 768 with a linear projection layer. Finally, we concatenate the end effector poses (20 Hz) from the past 2 timesteps to the audio-visual feature.Policy Learning. To model the multimodality intrinsic to human demonstrations, we choose to use a diffusion model with UNet encoders as proposed by Chi et al. [2] as the policy head, conditioned on the observation representation mentioned above in each denoising step. The entire model (Fig. 3 ), including the above-mentioned encoders, is end-to-end trained using the noise prediction MSE loss on future robot trajectories of 16 steps.Audio Latency Matching. During data collection, the vision and audio data are synchronized when recording through GoPro. During deployment, we calibrated the audio latency to be 0.23; details can be found in the appendix. We adopt an approach similar to Chi et al. [1] to compensate for this delay. 
evaluation: 
 We study four contact-rich manipulation tasks to show the different advantages of learning from audio feedback, such as detecting contact events and modes (flipping and wiping), or sensing object state (pouring) and surface material (taping). In each task, we test the policy under different scenarios and compare with alternative approaches to validate the robustness and generalizability of our approach.The robot is tasked to flip a bagel in a pan from facing down to facing upward using a spatula. To perform this task successfully, the robot needs to sense and switch between different contact modes -precisely insert the spatula between the bagel and the pan, maintain contact while sliding, and start to tilt up the spatula when the bagel is in contact with the edge of the pan.We collect two types of demonstrations for this task: 283 in-lab demonstrations and an additional 274 in-the-wild demonstrations collected in 7 different environments using different pans.Test Scenarios: We run 20 rollouts for each policy. To ensure a fair comparison, we use the same set of robot and object configurations for evaluations between different methods. We achieve the same object configuration by overlaying their position with respect to a captured image in the camera view. The test configurations can be grouped into four categories.• T1: Variations in task configuration: different initial robot and object configurations (14 / 20).• T2: Audio perturbation by playing different types of noises in the background (2 / 20).• T3: Generalization to unseen table height (4 / 20).• T4: Generalization to two unseen in-the-wild environments: a black stovetop and a white countertop, the latter is more challenging due to unstructured background and lack of similar training data.Comparisons: In this task, we focus on comparing our results with several ablations of the network design:• Vision only: the original diffusion policy conditioned on image observations. • MLP policy: using an MLP with three hidden layers (following Li et al. [33] ) instead of action diffusion. The model takes the observation representation and outputs the future action trajectory. [MLP fusion] policy often fails to fully wipe out the drawing and terminates early.• ResNet: uses a ResNet18 encoder to encode the audio log-mel spectrograms, with an additional CoordConv layer [43] following Li et al. [33] . • AVID: Following the approach by Mejia et al. [30] , use a 9-layer CNN audio encoder pre-trained on AudioSet [44] using Audio-Visual Instance Discrimination (AVID) [45] . • In-the-wild: For in-the-wild evaluation (T4), we compare our model trained with in-the-wild demonstrations (blue bar in Fig. 5 In-the-wild data enables generalization to unseen in-the-wild environments. As shown in Fig. 5 , policy trained on in-the-wild data significantly outperforms in-lab data on two unseen environments, as the scene diversity in in-the-wild data allows the policy to generalize better to new environments.In this task, the robot is tasked to wipe a shape (e.g., heart, square) drawn on a whiteboard. The robot can start in any initial configuration above the whiteboard and grasp an eraser in parallel to the board. The main challenge of the task is that the robot needs to exert a reasonable amount of contact force on the whiteboard while moving the eraser along the shape. We collect 119 demonstrations in total for this task.Comparisons: In addition to the vision only baseline, we evaluate the following alternatives for processing and learning from audio data:• MLP fusion: uses an MLP with 2 hidden layers to fuse the vision and audio features instead of a transformer encoder. This approach was used by Du et al. [31] . • Noise masking: without noise augmentation but instead mask out the audio frequency below 500 Hz, which is the UR5 control frequency. • No noise aug: without augmenting the audio data with noises during training.Test Scenarios: We run 20 rollouts for each policy. In addition to the T1 (5 / 20) and T2 (4 / 20) test cases as described above, we also test generalization to unseen table heights, erasers, and drawing shape T3 (11 / 20) . A detailed breakdown of the test scenarios can be found in the appendix. Findings: The quantitative result and typical failure cases are visualized in Fig. 6 . We find that: 1) Contact audio improves robustness and generalizability. Using only images from the wrist-mount camera, it is hard to infer whether the eraser is contacting the board or not, whereas incorporating the contact audio improves the overall success rate from 40% to 85%. The [Vision only] policy fails to generalize to unseen table heights and unseen erasers. To understand the behaviors of the vision only policy and our policy better, we visualize the attention map of the vision encoder in Fig. 4 and find that the model trained with audio attends better to task-relevant features. 2) Noise augmentation is an effective strategy to bridge the audio domain gap and increase the system's robustness to out-of-distribution sounds. Without noise augmentation, the robot is less robust to noises during test time and does not generalize well to unseen table heights, unseen erasers, and unseen shapes. Another alternative we study is simply masking out the robot noise regions. Surprisingly, this approach yields slightly better results than [No noise aug] by removing the domain gap caused by robot motor noises. However, this alternative does not successfully address other noises during the robot execution, as mentioned in Section 3.2. Lastly, we show 3) the advantage of using a transformer to fuse the vision and audio features compared to using an MLP. A typical failure in [MLP fusion] is that the robot fails to wipe in the area of the shape and stops when the shape is not completely wiped off.The robot is tasked to pick up the white cup and pour dice out to the pink cup if the white cup is not empty. When finished pouring, the robot needs to place the empty cup down to a designated location. The challenge of the task is that the robot cannot observe whether there are dice in the cup or not given the camera viewpoint both before and after the pouring action. We collect 145 demonstrations for this task, with a 'shaking' action to generate vibrations that can be captured by the contact microphone if there are dice inside the cup.Comparisons: In addition to the vision only baseline, we study how the length of the audio input affects the policy performance with two ablations: one using 1s history audio and one using 10s history audio.Test Scenarios: We run 12 rollouts for each policy. In addition to the T1 and T2 scenarios, we also tested generalization scenarios (T3) to an unseen number of dice (e.g., no dice or > 6 dice) and unseen objects (e.g., screws or beans).The quantitative result and typical failure cases are visualized in Fig. 7 . Our key findings in the experiments are: 1) Combining with information-seeking action, audio can provide critical state information beyond visual observations. As illustrated in the figure, the vision only policy fails to execute the pour action as it cannot infer whether there are dice in the cup or not, whereas the policy trained with audio feedback can leverage vibrations to infer the state information and generalize to objects with similar sounds (e.g., screws). 2) Policy performance is sensitive to audio history length. As shown in the result, [1s audio] yields significantly lower performance since the shortened audio window does not contain sufficient information to guide robot actions. [10s audio] shows that even though a longer audio window contains sufficient information, it increases the complexity of the learning process and requires a higher capacity model. The robot is tasked to choose the 'hook' tape from several velcro tapes (either 'hook' or 'loop') and strap wires by attaching the 'hook' tape to a 'loop' tape underneath the wires. We collect 193 demonstrations in total, with a 'sliding' primitive where we use the tip of the gripper finger to slide along the tape. We run 10 rollouts in total, each time the robot is presented 2-4 velcro tapes in random order with at least one 'hook' tape.Comparisons: In addition to vision only, we compare the following baselines: • Env Mic: Instead of using a contact microphone, we mount a Rode VideoMic GO II directional microphone on the GoPro camera for both data collection and deployment to collect the audio signals.• Noise Reduction: Instead of applying training time noise augmentation, we evaluated an alternative method that uses a test-time noise reduction algorithm. The algorithm estimates a noise threshold for each frequency and applies a smoothed mask on the spectrogram [46] . More details can be found in the appendix. Findings. 1) Contact microphone is sufficiently sensitive to different surface materials. As shown in Fig. 8 , the [Vision only] policy makes random decisions and yields a 20% success rate. Similarly, the system that uses environment microphone to collect audio data achieves similar results as the [Vision only] method, as it fails to pick up the subtle differences between the surface material. In contrast, by leveraging the contact microphone, our method is able to reliably guide the robot to pick up the correct tape. 2) Training-time noise augmentation is more effective than test-time noise reduction. This is because the noise cancellation algorithm causes the signal to be deprecated, resulting in a domain gap between training and testing. On the other hand, our noise augmentation method preserves the frequency distribution of the original signals. More visualization can be found in the appendix.On the data front, even though the contact microphone can pick up a wide range of audio signals and is robust against environment noises in the background by design, it may not be useful in scenarios where the interaction does not generate salient signals (e.g., for deformable objects such as cloth or quasi-static tasks), and can easily become imperceptible due to robot motor noises during deployment. On the policy front, the current method does not leverage the fact that audio signals are received at a higher frequency than images and can be used to learn more reactive behaviors. Future work can consider a hierarchical network architecture [47] that infers higher frequency actions from audio inputs. 
conclusion: 
 Audio signals reveal rich information about the robot interaction and object properties, which can ease the learning of contact-rich robot manipulation policies. We present ManiWAV, an in-the-wild audio-visual data collection and policy learning framework. We design an 'ear-in-hand' gripper to collect human demonstrations that can be directly used to train robot policies through behavior cloning. By learning an effective audio-visual representation as condition to the diffusion policy, our method outperforms several alternative approaches on four contact-rich manipulation tasks and generalizes to unseen in-the-wild environments. method is able to reliably guide the policy to pour objects and place the cup, whereas the baselines either never execute the pour action or the place action, resulting in low substep success rate. We also find that the policy can generalize to unseen objects such as screws. In Fig. 11 , we visualize the audio spectrogram when the robot 'shakes' the cup. We can observe that the spectrogram for an empty cup is distinctive from a non-empty cup, and a cup with screws generates a similar audio pattern as a cup with dice upon shaking. We hypothesize that the audio features for the cup with screws and the cup with dice are also close in the audio feature space, leading to similar policy behavior. Videos of the policy rollouts can be found on the project website. A breakdown of the success rate for each substep in the taping task is shown in Tab. 2. 'Touch' is successful if the robot slides along the tape while maintaining contact, 'Sense' is successful if the robot chooses the correct tape, 'Pick' is successful if the robot successfully grasps the tape. 'Place' is successful if the robot successfully places the tape on top of the wires. By leveraging audio feedback to infer the object surface material (whether the tape is a 'hook' or 'loop'), our method is able to reliably guide the policy to choose the correct tape, whereas the baselines make random decisions, as shown in the 'Sense' step success rate. Videos of the policy rollouts can be found on the project website.Noise Reduction Algorithm. We use the non-stationary noise reduction method introduced in [46] . The algorithm computes a spectrogram from the audio waveform, and then applies an IIR filter forward and backward on each frequency channel to obtain a time-smoothed version of the spectrogram. A mask is then computed based on the spectrogram by estimating a noise threshold for each frequency band of the signal/noise. Finally, a smoothed, inverted version of the mask is applied to the original spectrogram to cancel Using a [ResNet] and [AVID] audio encoder results in the spatula to 'displace' most of the times, likely because the model is not sensitive enough to the sound feedback of spatula touching the bottom of the pan, and as a result the robot keeps moving downward and causes the spatula to displace. 
b) contact modes a) contact events c) surface materials d) object states: 
 furry rough 
 
 Appendix:  


Decision: **Post-Rebuttal Guidance**

Recommendation: ACCEPT. All the authors unanimously suggested a weak accept. The only concerns the reviewers had was on novelty (minor improvement over prior work on UMI), however this was mitigated by both strong results demonstrating the importance of the changes and the author clarifications in the rebuttal. This will be a interesting paper for the CoRL community.

**Pre-Rebuttal Summary**

The paper proposes a new hardware (a UMI gripper augmented with a audio device) and approach to jointly learn from audio-visual signals.

Strengths:
- Paper is easy to follow, well written
- Excellent results demonstrated the usefulness of audio data
- Excellent real robot evaluation

Weaknesses:
- Over claimed contribution of the hardware, since the proposed hardware is a extension of UMI - authors please clarify this.

Reviews: review: title: title: The authors have studied how audio data in human demonstrations can aid in learning contact rich manipulation tasks. Although the work is very interesting, it lacks certain insights in design choices and does not convince why audio can be better than tactile data apart from the economic aspect
paper_summary: paper_summary: Problem Studied by Authors: The authors in this work explore the feasibility of adding audio as a sensing modality to learn manipulation skills. This incremental work extends the prior work [34] cited by the authors in references. It is interesting to see how audio can help understand object properties (e.g., furry, rough, etc) and thus provide rich contact information. On a very high level, the authors have integrated audio and demonstrated how it can improve manipulation capabilities in addition to other modalities. Proposed Solution: Building on prior work, the authors introduce audio sensing to learn richer representations of manipulated objects. Their main contribution is leveraging audio to aid in learning a diffusion policy for executing contact-rich manipulation tasks.
main_review: main_review: The authors in this work have demonstrated how to encode audio signals and fuse them with vision features to learn a diffusion policy that can predict actions to perform manipulation tasks. The evaluation by the authors provides clear arguments on how exactly audio signals help. However, there are some clarifications required in certain design choices and implementation details of the system.

Strengths:
1. Adding audio as a modality in manipulation. Specifically, understanding object properties and learning richer representations.
2. The system is simple and easy to understand
3. The data augmentation strategy for audio data is interesting
4. The cost of implementation is low for human demonstration data collection

Weaknesses:
1. The authors claim that audio data is better than wrench/tactile data. However, they have not performed any evaluations to support this claim.
2. Furthermore, the use of wrench data has been extensively studied in manipulation tasks. Yes, tactile data can be expensive to collect demonstrations, and audio data can serve as a useful, cheaper alternative; however, there needs to be some comment on the degradation/upgradation in performance by replacing tactile with audio.

General Comments:
1. Claiming any advances on the hardware front would be a stretch since the UMI design was not significantly updated to account for the acoustic sensor. 
2. Fig. 4 is very interesting.

Writing:
1. The paper is well-written and easy to follow. The authors highlight the key contributions clearly. 
2. Very minor typos are present in the manuscript. I’d recommend running it through spellchecks.
3. The appendix is helpful in understanding the engineering difficulties and other ablations.
questions: questions: 1. Can the authors provide more insights into the design choices? Why use CLiP-pretained ViT? Specifically, when fusion is happening on audio data and ViT? Why not use something similar to Sung-Bin, Kim, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. "Sound to visual scene generation by audio-to-visual latent alignment." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6430-6440. 2023. In this work, the authors have shown how one can align audio embedding with visual embeddings to generate visual scenes from audio. It is understandable that CLIP-trained ViT might give sufficient performance, but some insights might be useful for the reader
2. Can the authors provide some evidence through preliminary experiments or prior work that the performance would be at par or comparable if wrench data was used? 
3. Would the intensity of background noise affect the performance?
4. Have the authors given a thought to adding something like a GelSight sensor to the fingers? Specifically, are you placing the sensors where the black-colored high friction pads are present? This increases cost but might give better performance (This is more of a clarification question)
5. Can the authors provide how sensitive the policy is to the number of demonstrations? Can using audio improve the data requirement for training diffusion policy?
limitations: limitations: 2
review_summary: review_summary: Overall, this work is compelling and highlights the significant role of audio in enhancing the learning of contact-rich manipulation policies. Building upon UMI, it introduces an innovative approach for collecting human demonstration data in real-world environments. However, further clarifications are needed to address some gaps in the understanding of the paper.
title: title: High potential impact with extensive experiments and excellent results but weak novelty
paper_summary: paper_summary: This paper proposes an 'ear-in-hand" demonstration device for teaching robot manipulation skills. As they show, audio can be used as a cheap and scalable alternative for rich tactile information. In contrast to prior work, demonstrations can be carried out without the presence of the robot or any augmentation to the environment. This is made possible through their policy learning method which utilises data augmentation to focus the policy attention on task-relevant portions of the audio.
main_review: main_review: # Quality/ clarity

The paper is well written and easy to follow. The figures and video are helpful in understanding the main ideas of the paper. Some more details on experimental setup could be provided (see rebuttal questions). Some hardware details in the method section could be moved to the experimental setup.

# Originality

While the idea of using audio is interesting it has been demonstrated in previous work. The main novelty comes from the data augmentation strategy which by itself is not majorly novel. This could be made clearer in the introduction.

# Significance of this work

Being able to demonstrate tasks in the wild is very useful and has high potential impact in the field. The demonstrated experiments showed significant improvement over baselines.

# Strengths

- Extensive experiments carried out with real hardware in various settings
- Results show significant improvement over baselines
- paper well presented, easy to read and understand

# Weaknesses

- no major theoretical or algorithmic advancement
- Some details missing/ need clarification (see rebuttal questions)
questions: questions: - What kind of background noise was present in T2? A more detailed study on the policy's robustness to different noise would interesting. For example, people talking, music, TV, etc... What happens if the noise is too loud and audio becomes unreliable?
- Can you provide more details on what the difference is between in-lab vs in-the-wild data/demonstrations are?
- In the wiping/ pouring tasks, it's not clear to me why the policy is able to generalise to unseen shapes/ objects. Can you provide more discussion on this?
- Regarding the pouring task, "the vision only policy fails to execute the pour action as it cannot infer whether there’s dice in the cup or not". Can you provide more details on why this occurs? I would imagine that even a vision based policy should eventually learn the correct pouring behaviour. 
- Is the sensor fusion transformer pre-trained?
limitations: limitations: 2
review_summary: review_summary: While this work has high potential impact in the field the novelty of the method is quite weak.
title: title: ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data review
paper_summary: paper_summary: The authors present a variation of UMI system with an embedded contact microphone. They use this handheld system to collect human demonstrations in-the-wild and present an end-to-end policy across four tasks (flipping, wiping, pouring, and taping). Within each experimental rollout, they define variations and ablations to compare the overall effect of their design choices such as data augmentation methods, noise filtering, or model architectures.
main_review: main_review: The authors present a variation of UMI system with an embedded contact microphone. They use this handheld system to collect human demonstrations in-the-wild and present an end-to-end policy across four tasks (flipping, wiping, pouring, and taping). Within each experimental rollout, they define variations and ablations to compare the overall effect of their design choices such as data augmentation methods, noise filtering, or model architectures.

Overall, the paper is well-written and experiments are well described and informative. The main strengths include four interesting demo tasks with detailed analysis and discussion. I appreciate the combination of in-the-wild data collection with diffusion policy, which helps support that their policies can generalize in different environments. 

My biggest concern is the consideration of the UMI interface+contact microphone as a main (1 of 2), novel contribution of this work. I believe the contact microphone is a very minor change, and does not warrant being listed as new proposed "portable hand-held device". I believe this is also supported by the amount of text spent describing the previous UMI device, and one additional sentence describing the contact microphone. With the above context, I find the work interesting to robotics community but with minor contribution compared to previous work.

The video supplement is nice addition to the work, covering a nice variety of sample audio+video from their data collection. It covers some views both on the human and robot side using the UMI interface. The videos also appear to be real-time, which helps ground the reader in what the final policies will look like.
questions: questions: The portable system presented in the first paper contribution bears a strong resemblance to the UMI interface, as acknowledged by the authors through citation. The soft finger structure, tape, GoPro, and markers appear to be identical. Beyond the addition of a contact microphone, are there any other significant differences that make this a primary contribution of the work?

Why did the authors choose to resample from 48kHz to 16kHz? Can you also clarify if this means you filtering out the higher-frequency data or downsampled to ease the computational cost?

For those working with tactile audio, I believe they would benefit from additional reasoning for the selection of window/hop length and other preprocessing decisions near line 146.

For each of the four tasks, there are about 100-120 demonstrations. How was this number decided? Was their any motivation behind why some tasks required additional demonstrations? I would be very interested to see an ablation across number of demonstrations required. While your device makes the data collection process efficient, I believe it is still valuable to understand the data efficiency of this approach. 

Similarly, why did the authors collect 200 demonstrations for the taping task? I am guessing 100 with the original contact microphone and 100 with the Rode microphone, but I did not see that distinction in the text. 

I found the summary of findings at the end of each section quite helpful and well-written. However, the Figures 5-8 were difficult to follow. Specially, Figure 5 bar charts are too small to read easily, and the overlaid text is hard to follow. Consider editing the caption to provide more context and moving some content to the appendix to emphasize your main findings.
limitations: limitations: 3
review_summary: review_summary: Despite similarities to previous work, I support a weak accept as the updated model architecture and data augmentation appear to have consistently improved results, which will be of interest to the community.