DiffImpact: Differentiable Rendering and Identification of Impact Sounds
Authors: Samuel Clarke
Negin Heravi
Mark Rau
Ruohan Gao
Jiajun Wu
Doug James
Jeannette Bohg
Venue: CoRL2021 Oral (2021)
Keywords: Differentiable Sound Rendering
Auditory Scene Analysis

Abstract: Rigid objects make distinctive sounds during manipulation. These sounds are a function of object features, such as shape and material, and of contact forces during manipulation. Being able to infer from sound an object's acoustic properties, how it is being manipulated, and what events it is participating in could augment and complement what robots can perceive from vision, especially in case of occlusion, low visual resolution, poor lighting, or blurred focus. Annotations on sound data are rare. Therefore, existing inference systems mostly include a sound renderer in the loop, and use analysis-by-synthesis to optimize for object acoustic properties. Optimizing parameters with respect to a non-differentiable renderer is slow and hard to scale to complex scenes. We present DiffImpact, a fully differentiable model for sounds rigid objects make during impacts, based on physical principles of impact forces, rigid object vibration, and other acoustic effects. Its differentiability enables gradient-based, efficient joint inference of acoustic properties of the objects and characteristics and timings of each individual impact. DiffImpact can also be plugged in as the decoder of an autoencoder, and trained end-to-end on real audio data, so that the encoder can learn to solve the inverse problem in a self-supervised way. Experiments demonstrate that our model's physics-based inductive biases make it more resource efficient and expressive than state-of-the-art pure learning-based alternatives, on both forward rendering of impact sounds and inverse tasks such as acoustic property inference and blind source separation of impact sounds.

DiffImpact: Differentiable Rendering and Identification of Impact Sounds
Negin Heravi Doug James Stanford University {spclarke, nheravi, mrau, rhgao, jiajunw, djames, Ostanfoodd edu
Samuel Clarke
Mark Rau
Ruohan Gao
Jiajun Wu
Jeannette Bohg
Abstract: Rigid objects make distinctive sounds during manipulation. These sounds are a function of object features, such shape and material, and of contact forces during manipulation. Being able to infer from sound an object's acoustic properties, how it is being manipulated, and what events it is participating in could augment and complement what robots can perceive from vision, especially in case of occlusion, low visual resolution, poor lighting, or blurred focus. Annotations on sound data are rare. Therefore, existing inference systems mostly include a sound renderer in the loop, and use analysis-by--yntheiis acoustic properties. Optimizing parameters with respect to a non-differentiable renderer is slow and hard scale complex scenes. We present DiffImpact, fully differentiable model for sounds rigid objects make during impacts, based on physical principles of impact forces, rigid object vibration, and other acoustic effects. Its differentiability enables gradient-based, efficient joint inference of acoustic properties of the objects and characteristics and timings of each individual impact. DiffImpact can also be plugged in as the decoder of an autoencoder, and trained end-to-end on real audio data, so that the encoder can learn to solve the inverse problem in a self-supervised way. Experiments demonstrate that our model's physics-based inductive biases make more resource efficient and expressive than state-of-the-art pure learning- alternatives, on both forward rendering of impact sounds and inverse tasks such as acoustic property inference and blind source separation of impact sounds. Code and videos are at sttps://sites google... cwm/v coon///eedddffimpaeae/eee
optimize for object
Keywords: Differentiable Sound Rendering, Auditory Scene Analysis
Introduction
The sound we perceive during rigid object contact a function of many factors, including noise in the environment, position of the listener, reflecting surfaces in the surrounding environment, etc. The most defining of these factors are the vibrations the object makes, a product its intrinsic acoustic properties and the way it was contacted. Extracting information about the acoustic properties of an object could inform a robot about the object's size, shape, material, and where and how was struck. Extracting information about the way the object contacted could inform robot of the velocity, mass, shape, and hardness of the tool that was used to strike the object. Therefore, the ability to interpret impact sounds could augment and complement what robots perceive from vision, especially in cases of occlusion, low visual resolution, poor lighting, or blurred focus.
Decomposing the sound an object makes into the contact forces that excited it and the idealized acoustic impulse response is generally an ioo-po problem. As annotations on sound data are rare, existing inference systems mostly include a sound renderer in the loop, and use mallyis-bb-synthese to optimize for object acoustic properties. Optimizing parameters with respect a non-differentiable renderer is slow and hard to scale to complex scenes. We propose Difflmpact, a fully differentiable, generative sound model that uses physics-based priors as a bias. Specifically, we introduce fully dif- ferentiable models of the object's impulse response and the contact force during impact. We combine these with differentiable models of environment acoustics such as ambient noise and reverberation into fully differentiable generative sound model. With this physics- based model, physically interpretable parameters of rigid body impact sounds through analysis-br bisiis-b--yynnssiii even
now infer
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

from real-world recordings, including from real YouTube videos or from noisy and reverberant robotics lab.
Impacts in the real world are not always clean impulses, nor do they occur in controlled acoustic environments. We show how Difflmpact's differentiability allows us to learn meaningful sound models that are fit to the sounds of multiple imperfect impact sounds over one or multiple recordings. We first validate our approach on single impact sounds. We show that our model can fit interpretable physical parameters of an object's sound model and of the contact forces to audio recordings. We then show how Diftlmpact enables robot to passively learn object sound models from wealdw dataset (ASMR YouTube videos [1]). Our model not only extracts physically interpretable parameters, but also outperforms model- and model-free learning baselines in rendering realistic impact sounds for held-out data. Finally, we show how a robot can learn object sound models from data it collected by striking these objects with a tool. Specifically, we show how the robot can perform blind source separation of the collected sound and thereby achieves 140% higher accuracy on classifying the material of the objects it strikes.
2 Related Work
Audio-frequency vibrations have been shown to be an effective modality for robots to ascertain characteristics of physical events in their environment, including instance level classification of everyday objects [2, 3] and estimates of their motion [4], terrain classification by legged robots [5], masses of granular materials being poured from a scoop the robot is holding [6], and amounts of liquid poured by a robot [7]. Auditory scene analysis also has applications for dynamic inference of stochastic events, enabling creation of audio based reactive robotic trackers [8].
Many of these works have a common characteristic with more recent visual perception approaches, that learning-base based neural methods are subsuming many classical approaches by enabling automatic feature extraction rather than requiring hand engineered features [9, 10]. However, whereas differentiable image renderers have empowered new approaches in visual perception [11, 12, 13, 14), differentiable audio renderers have been introduced only recently. Engel et al. [15] recently debuted the Differential Digital Signal Processing (DDSP) library, showing applications in the musical domain on tasks such as disentangling the pitch and timbre of instrument sounds. Rather than the musical domain, we focus on modeling modal impact sounds of rigid objects, and show applications of our mrnysics-based model for robotics tasks.
Model-free neural audio representation methods powerful in synthesizing realistic human speech [16, 17] but lack interpretability and flexibility for transfer to different categories such as transient environment sounds. Model-based methods often require less data and training time to produce interpretable learned parameters, which can be reused and applied to other tasks more intuitively. They are also capable of synthesizing realistic object sounds [18, 19, 20] but depend on sand-engineered analytical methods for parameter estimation with structured audio samples. Ren al. [21] use real audio clips of striking objects infer properties such as Young's 's modulus and the modes of the object, then attempt to reproduce the sound based on a modal model with a residual. However, they assume knowledge of the 3D geometry of the object as well as impact location. Zhang et al. [22] infer object properties such as Young's 's modulus and Rayleigh damping from sounds of objects falling on a surface by using a physics-based modal sound generation engine with a black box forward model. But their model is not fully differentiable, making it challenging to learn from unconstrained in-the-wild audio or to optimize over many dimensions at once. While we use a model conceptually similar to that of [23, 22], our fully differentiable structure empowers learning a best estimate of an object's modal model from the sound of multiple uncontrolled, imperfect, heterogencous impacts, rather than requiring strictly controlled recordings for analysis.
3 Differentiable Rendering of Impact Sounds
Our DiffImpact models the sound that a rigid object makes when struck in a real environment as a function of 1) the impact force's 's magnitude over time; 2) the brief click of a sound pressure wave caused by the rapid deceleration of the striking object, referred as the "acceleration sound"; 3) the struck and striking objects' impulse responses, or their idealized theoretical vibrational response to a perfect unit impulse; 4) any environment, background, or recording noise; and finally 5) the impulse response of the local environment, reflecting and transmitting reverberations of any sound being directly emitted by the object. DiffImpact attempts to decompose real impact sounds into each of these contributions, using physics-based models in order to regularize this decomposition.

---

GT Spectrogram
Generators
End-to-End Learning Dataset
Force Profile
Impact
Force
Final Waveform
Medal Responso(s)
Encoder
Modal
Impact
Source Separation
Background Noise
Filtered Noise
Object
Spectrogram
Enw Response
Reverb
Reverb
Parameters
L1 Loss
Figure 1: Overview of proposed approach, Difflmpact. Blue elements are only used for the end-to-end learning task, yellow elements are only used in source separation, and green elements are used in both. Dashed components are only used during training, and elements with red outline have learnable parameters.
3.1 Overview
Figure provides a high-level overview of our approach. There are some model variations dependent on which inference task are considering. For clarity, here will give overview of the model when used for the end-to-end learning task of predicting an audio waveform from a magnitude spectro- gram of the ground truth audio sample, similar the models [15, 24, 16]. The magnitude spectro- gram lacks phase information and is therefore not directly invertible to an audio waveform. This spec- trogram is input to an encoder network that outputs coarsely temporally discretized time series pa- rameters that represent impact characteristics. These parameters are the input to the Impact Force Pro- file Generator that synthesizes an impact force profile as described more detail in Section 3.2. Given object-speciffe learnable parameters of modal model of rigid body impact sound, the Modal Impudse Response Generator synthesizes an impulse response as described more detail in Section 3.3. This impulse response is convolved with the force profile to synthesize object vibrations. To model the contribution of the sound pressure wave caused by the rapid deceleration the impacting object, this impact force profile is also multiplied by learnable scalar and added to the object vibration sounds.
We theoretically now have all the sounds emitted directly from the point of impact on the object. To model sound that is specific to the environment, we introduce a noise and reverb generator as detailed in Section 3.4. Given learnable noise parameters, the Filtered Noise Generator synthesizes recording or background noise that is then added to the current audio. Finally, given learnable reverb parameters, the Reverb Generator synthesizes an impulse response of the environment. This response is then convolved with the current audio to model the sounds that have transmitted through the environment and reflected off nearby surfaces, rather than directly from the object. This result added to the current audio to produce the final waveform.
For training the aforementioned learnable parameters of the generators and the encoder, we use a loss that compares spectrograms of the generated waveform with those of the ground truth (Section 3.5).
The final waveform (t) produced by our model can be expressed as
(Ft)=[F((t,,t,77) @ @ Ia+
(1+IRe
(1)
where is the time in the final waveform; F' is the force profile function that takes as input the impulse magnitudes m, impulse timings and contact time scales T; @ denotes convolution; a is scalar for the acceleration sound; IR, is the modal object impulse response function parameterized by modal frequencies f. gains go. and dampings do and fıR is time within each impulse response; N is the filtered noise parameterized by frequency band gains gn; and I IR, is the impulse response of the environment parameterized by frequency band gains ge and dampings de We will now detail the equations and physics- models for each of these components we propose, and how we scale a low dimensional set of normalized parameters into input parameters for each of these functions, producing an audio frequency signal in differentiable manner.
3

---

3.2 Differentiable impact force profile and acceleration sound
To model the force profiles of multiple impact events occurring over the time interval of an audio clip. our proposed module converts the two time series outputs from the encoder into contact force profile at audio frequency. This profile is defined by the time scales T and magnitudes m of the impulses.
First, we define a physical model for each impulse event. Since objects will generally have convex curvature the points at which they strike, they can be locally approximated as spheres in contact. Therefore, we adopt Hertz half-sine contact model based on collisions between frictionless rigid spheres similar to that of [25]. To ensure smoothness, we use a Gaussian approximation C₁ (4;42,TT)) of this contact model force impulse profile (26, 27]:
6 (1-4 (e: (1-4--)
C((((mm))=xpp
where f represents the timing of the onset of the unsmoothed impact, and Ti is the time scale of the contact force, which depends on the velocity of the impact and material properties of the object [28]. Sharper and faster contact points as well as harder materials will each contribute to reducing the value 7r,
Ti, while softer, slower, and duller contact points will have higher 7, values.
Equation 2 provides model for an individual contact event. But in a natural audio clip, multiple such contact events, each with different characteristics, can occur over given interval, such that natural force profile over time will be the sum of multiple such contact impulses, each with different parameters and magnitudes, and each occurring at different instants during the interval. We model the force profile, the scalar value of the normal force over time, for multiple contacts as a weighted sum of such Gaussian impulses from Equation 2:
F(tm,,;,)=mi C(t,t,T) (((ttt))
(3)
where t [Mo T= (To.....Ta]]] ]" and m- weight for the corresponding impact. C(t,t,T) stacks the output of Equation 2 into a vector.
[mm....,,ar]]] I-
Each magnitude m, serves as the
Equation 2 is naturally differentiable, but a challenge making Equation 3 fully differentiable is that the number of impacts occurring during an interval may not be be known a priori, and may vary be- tween examples. To address this, we propose algorithm that finds M peaks in time series of impact magnitude estimates over time, and places model-based impulses at those peaks based on the param- eters each peak's point in the time series. If there are fewer than M peaks, the impact magnitudes my at the extraneous peaks should be estimated as zero. We detail this algorithm in Appendix A.1.
To model the acceleration sound, we must know the acceleration profile of the striking tool's tip. Since the striking tool's mass is fixed, then the deceleration is proportional to the impulse force, which is already estimated in Equation 3. Therefore, we model the contribution of the acceleration sound as the force profile from Equation 3 multiplied by the learnable scalar variable a.
3.3 Object impulse response
For the object impulse response, adopt modal model by modeling objects drring-damper systems, where the impulse response is a linear combination of exponentially decaying sinusoids, with each sinusoid having a different frequency Sk. gain yk, and damping de [18, 23]. We model the K most salient modes of the object through constructing a finite impulse response (FIR) as [t; go: do) go [exp(-do1) (ooodddd))sssnnntfde d. =d=ldo.....d the gains of each object are fixed across the entire body of the object, whereas in reality for general objects, they vary significantly depending on the location at which the object is impacted. We made this assumption make the optimizations in our experiments more tractable since the inputs to our frameworks were blind to the 3D models of the objects as well as the locations of each contact. We detail how we scale normalized inputs to our module to produce the parameters in Appendix A.2.
where f [So ... fK). go -00.---9KK and o is the Hadamard product. Note that we make the simplifying assumption that
3.4 Environment noise and reverberation
We approximate background and measurement noise as static filtered noise, whereas we approximate reverberation from the measurement environment with exponentially decaying filtered noise. For background and measurement noise, our filtered noise function N(lgg.) filters random white noise with time-constant gains for each frequency band given by learnable noise parameters: N(I:gn) = gn T filt(N((t), Bn), where (t) B,,) E R RB.x7 XT is a function that filters white noise N(t) with

---

Force
Time
Time
Time Ground Truth
Time
Ground Truth
Diffimpact (ours)
Diffimpact (ours)
Figure 2: Comparing our model's estimates to ground truth time series of normalized contact forces and spec- trograms, for our analysis by synthesis task short recordings. Results are from the steel bowl.
BB band pass filters spaced linearly over the interval of frequencies to the Nyquist frequency. The output matrix is multiplied with the vector ga weighting each frequency band.
For generating reverberations, we model the local environment's 's response to sounds with an impulse response with exponentially decaying filtered noise, where each noise band has its own decay rate, similar to [23]: RRe (LrRige, de), ge [exp(=dal) (V)) and damping factors per frequency band, respectively, and have dimension Be. We describe how these parameters are derived from scaling our learnable normalized reverb parameters in Appendix A.3.
Here ge and de are the gains
3.5 Loss function
As loss function for all tasks, we use the sum of the L1 distance between the spectrograms and log spectrograms of multiple window sizes from the ground truth W and synthesized audio W [15, 24]:
L(W,W)=
Σ
(w,, S(W SWW,,,WWWW,... ,s,, | (Sw.llog(((W,,wii..)l.
(SW ,.m,BBw))
*w E(18........ 2048)
where sw is the window size, B is the overlap ratio, A and 22 are weights, and S is the short-term Fourier transform (STFT) or spectrogram. Using multiple spectrogram scales ensures that the loss characterizes eπrоr signal in both high frequency and temporal resolution. The log spectrograms also ensure that higher frequency components, which tend to have lower energy magnitude than low frequency components, are also weighted more strongly. We simultaneously fit estimates of param- eters for both the sound the object makes when impacted, and the impacts that produce those sounds.
Results
We first validate that Difflmpact is able to not only learn expressive sound models, but also to extract accurate physically interpretable parameters from controlled recordings of impacting different house- hold objects. Then we show how our contributed modules can be used in two different real-world tasks, one showing how a robot could use our models to passively learn sound models from data, and another showing how our robot can interactively learn sound models and use them for useful downstream tasks.
4.1 Analysis by synthesis: Fitting to single real impact sounds
To test whether our DiffImpact can extract contact forces from sound, we collected 3-second clips of striking four different household objects (bottom right of Figure 4) in recording studio with an impact hammer (PCB 086C01). The impact hammer had a force transducer in its tip. providing ground truth contact forces synchronized with the audio recorded by microphone (PCB 378A06). We randomly initialized the parameters for our model, and fit them to each recording with respect to our loss function which only compares synthesized and ground truth spectrograms. We use Adam as the optimizer [29]. Our results are shown in Figure 2. Though our model's estimates of contact force have no sense of scale or calibration to physical quantity (see Appendix E), we show that the force profiles our model synthesized were quite similar in shape to those of the ground truth. Note that do not supervise directly force profiles. Furthermore, the spectrograms demonstrate that our model is able to synthesize audio very similar to the ground truth. Audio examples are included in the Supplementary Materials, and further details of this experiment and its setup are in Appendix B.
4.2
End to end learning: Learning rigid body impact sound models from in-the-wild audio
We demonstrate the utility of our modules in the end to end learning task of autoencoding real world impact audio. Similar previous work [16, 15], the input to each model is the ground truth magnitude spectrogram, and the output is the wave form of the audio. Our experiments in this section driven by evaluating perceptual realism and similarity of our model's predictions, measured both qualitatively by human user studies and quantitatively by perceptual loss metrics.
Dataset We curated a dataset of audio clips extracted from videos on the ASMR Bakery YouTube Channel [1]. We selected clips from segments in which the creator is repeatedly tapping five objects
5

---

% Preferring Our Methed
100
---
Ground Tuth
Breateven
1.0
Wh te Noise
80
DOSP
Normalized Distance
D Diffimpact (ours)
DDSP Serra
Wineoloow
DDSP Sin
0.6
WaveGlow
40
0.4
20
0.0
Wonden Box Glass doel
Siloone Aad
Steel Bar
Caramic Planc
Average
CDPAM
Spectral
Envelope
Figure:: 3:
Results from end-to-end learning task evaluations on ASMR dataset. (Left) Human study ratings of
realism of different ASMR objects' generated audio from test (Right) Average computational audio distance metrics of test set output from different models.
of different materials: wooden box, glass bowl, silicone pad, steel bar, and ceramic plate. Screenshots from clips of each of the objects we selected are shown Appendix C.I. With this dataset, we show that a robot could use our framework to learn meaningful, physically interpretable models of rigid objects from a rich real world dataset. This dataset presents many challenges to the task of estimating modal sound model, challenges which prior works have not addressed. First, whereas other works record the sound of a single controlled impact by a small pellet or specialized instru- ment [23, 18), many clips in our dataset include impacts randomly alternating between tapping with hard fingernails and soft fingertips within short timespans. Some of these objects are being held in the creator's i hand while they are being tapped, and other objects are lying table (see Appendix C. Holding each object or resting it on a surface will dampen certain modes of vibration. Therefore, no point in the entire dataset does object emit an ideal undamped modal response. The ASMR dataset consists of around 3-5 minutes of continuously recorded audio of tapping for each object. We validated our model and its hyperparameters remaining data for each object in approximately a 90-10 train-test split. Thus, each model had 200 seconds of training data for each object, and at least 20 seconds of unseen audio samples per object.
held out validation set of the steel bar clips, then split
Baselines We implemented two baseline models based on DDSP [15, 24], which we refer to as "DDSP Serra" and "DDSP Sin." DDSP Serra is based on a time-varying harmonic model developed for musical instruments [30]. To test the influence of the harmonic model bias, DDSP Sin replaces the harmonic oscillator with unrestricted time-varying sinusoidal oscillator. We also tested WaveGlow [16]. a state the art model-free approach for generating human speech. Further details on baselines are in Appendix C.2. The WaveGlow baseline took more than two orders of magnitude more GPU time to train (~~1 hour vs. ~ 300 hours of training time for a single object on an Nvidia Quadro P5000 GPU). To demonstrate the necessity of this training time, we report results from training this model for only ~ order of magnitude more than the other models in Appendix C.4. For our evaluations, we trained a separate instance of each baseline and our model per object.
Qualitative evaluation through human studies To test Difflmpact's capability of producing real- istic impact sounds on held-out audio samples of each object compared baselines, re-dubbed the original videos with each model's output given the video's 's original audio as input. We then presented participants on Amazon Mechanical Turk with a two-alternative forced choice to pick the best match- ing audio for 5-second videos. The comparisons were between our model and either the baselines or ground truth (1200 questions total). Our model significantly outperforms all the baselines on average (p<00.05) as shown on the left of Figure 3. Per object category, DiffImpact outperforms all baselines for the steel bar and ceramic plate. It outperforms DDSP Serra and DDSP Sin models for all objects except for the silicone pad (p<00.5). We suspect our model struggles with modeling the impact sound of the silicone pad due the damping effect and roughness of the material as well as its high deforma- bility that violates our model's rigid body assumption. The other models without these biases may better model its sound. Furthermore, the user's hand sometimes rubs against the surface during con- sequent tapping. resulting in unmodeled components in the audio which affect the performance of our method. On average, our participants chose our model's 's audio over ground truth 42.9% of the time, suggesting that our generated audio is often realistic enough to be confused with ground truth audio.
Quantitative evaluation using distance metrics We also compared each model's test outputs with the ground truth audio using different computational distance metrics, including CDPAM, learning- based perceptual distance metric designed for human speech applications [31], the multi-scale spec- trogram loss used in [15, 24], and the envelope distance used in [32]. Since each of these metrics have

---

Figure 4: Physical setup and tools for rohotic impact dataset collection. (Left) Franka Panda holds and rotates a ceramic chopstick to swing at the wood napkin holder, with audio recorded by the bottom-left microphone. (Middle) Close up of a fork swung at a mug. with microphone at left. Objects were rested on a soft foam pad to prevent sliding and mitigate contact damping and acoustic reflection from the hard tabletop. (Right top) The tools used for striking (from top): wooden spoon, polycarbonate spoon, ceramic chopstick, and steel fork. (Right bottom) The struck objects (from left): steel bowl, wood napkin holder, ceramic mug, and polycarbonate cup.
widely different scales, we normalized each model's 's average distance on each metric by the average distance of white noise with respect that metric. The results are shown the right of Figure 3. These results align with our human studies, suggesting that our DiffImpact outperforms DDSP Serra and Sin, and it performs on par with WaveGlow, yet requires two orders of magnitude less training time.
Discussion We demonstrate that Difflmpact is expressive enough characterize and generate real world impact sounds, with performance competitive or better than state of the art alternatives. But perhaps most importantly, while completing this end-to-end learning task, our model forced to extract physically interpretable and transferable parameters of impact forces and acoustic properties of the object and environment, whereas the parameters learned by other models are either uninter- pretable (in the case of WaveGlow), not directly meaningful to extracting impact event and intrinsic acoustic properties of objects. These parameters can be used with a novel impact profile to render the sound the object would make when impacted as such, creating multimodal object model for simulation environments [33, 34] merely from information learned from in-the-wild Yuueube videos.
4.3 Robotic task: Source separation of kitchen utensil sounds
We use DiffImpact to perform a source separation task. Unlike traditional audio source separation, where the goal is to isolate sounds in passive, pre-recorded audios and videos [35, 36, 37], our goal is enable robots to actively interact with objects in the environment and separate the sounds they make. Consider the scenario where the robot strikes an object with a tool. The perceived sound of the impact will be combination of the sounds the tool makes and the sounds the struck object makes. We show that by separating them, a robot can more accurately classify object materials, as different materials have distinct acoustic properties. Knowing the material of object can inform the robot about its durability, surface friction, and category, all key factors to consider when planning grasps and actions.
Dataset We collected dataset of a robot swinging four houschold tools ("tools") at each of four household objects ("objects") of four different materials, with each distinct material represented in exactly one element of each category. We recorded each impact with Shure SM57 microphone. Photos of the robotic setup and the tools and objects are shown in Figure 4. We filtered out any samples where the recording was too loud and clipped, which would obfuscate the frequency content, then used the two loudest remaining recordings of each pairwise interaction for our experiments, to ensure high signal noise ratio.
Optimization setup We used the yellow elements of the model shown in Figure 1. Compared to the description in Section 3, we modified the model as follows: I) instead of using the neural encoder control impact force, the impact force profile for each instance was single impulse parameterized by the Impulse trainable parameters, and 2) the Tool parameters were also used to produce a modal impulse response which was convolved with the force profile along with the Object modal impulse response. The parameters for each tool and each object were optimized with respect to each of the recordings for which they were present. All recordings shared the same set of Noise parameters, with each having single gain scalar for the noise. We then performed analysis by synthesis on the entire batch of 32 recordings at once, optimizing all our trainable variables with the Adam optimizer with respect to the loss that only compares ground truth with generated spectrograms of the non-separated
7

---

sound. For producing the separated object sound, we turned off the background noise, reverberation, acceleration sound, and the tool impulse response, then generated the model output for each recording. For producing the separated tool sound, we turned off the background noise and the object impulse response, then generated the model output for each recording. See Appendix D) for more details.
Baselines We compared with non-negative matrix factorization (NMF) clustering [38] and the more recent "Deep Audio Prior" (DAP) [39] networks. Similar to our framework, both basclines require no pre-training. Note that these baselines do not have built-in way of specifying which separated source corresponds to the tool and which to the object, whereas ours does.
Metrics We provide qualitative examples of the separated sources in our Supplementary Materials. Ground truth separated object sounds are not available to compare our results against, as there is no realistic way to even collect them in the real world. Thus, for a quantitative comparison, we compare the performance of material classification on each framework's 's outputs. Specifically, we train an audio-based material classifier on balanced data of only the relevant materials from the Sound-20K dataset, which consists of only synthetic sounds of objects of different materials bouncing on surfaces in a virtual environment [40]. We tested this trained classifier on the original sound as well as the source-separated output of each framework, to test whether the separated audio can be more accurately classified by this trained material classifier.
Table I: Classifying materials from source-separated audio.
Results report results in Table 1. Because our baselines have no con- straint which separated source is the tool and which is the object, we
Object Material Ace. (%) Tool Material Acc. (%)
Model
Dataset Max
Inst. Max
Dataset Max Inst. Max
show the results of two evaluation schemes. Under the "Dataset Max" scheme, we compute the object mate- rial accuracy twice for each baseline: first treating their first output as the object, then treating the second as the object. We use the higher between the two as their result. We do the same for tools. Under the "Inst. Max" scheme, we take the maximum for each instance instead of across the whole dataset. While both schemes offer the baselines an advantage, our Difflm- pact still outperformed them, while also explicitly differentiating between the object and the tool. All models, including ours, performed better for classifying objects than tools. Object sounds dominated the recordings over tool sounds, since they were much larger, with more surface area for dispersing vibrations. Their size also gave them stronger low frequency modes with lower dampings, lasting longer in recordings, and thus easier to capture and characterize. See Appendix D for confusion ma- trices from this task, as well as results from similar experiment where our DiffImpact outperforms the same baselines for estimating Young's 's modulus of each object and tool from separated sounds.
Raw audio
37.5
37.5
12.5
12.5
NMF [38]
40.6
43.8
21.9
21.9
DAP [39]
50.0
68.8
28.1
50.0
DiffImpact (ours)
90.6
96.9
43.8
53.1
5
Conclusion
We have proposed DiffImpact, fully differentiable physics- based framework for modeling rigid ob- ject impact sounds. This model enables robots to learn valuable physically interpretable parameters from sounds of objects in their everyday environments. We first validated that our method success- fully simulates impact sounds of different everyday objects, and that it can be used in an analysis-by. synthesis approach estimate impact forces that generated the analysed sound. We also showed how our framework can be used as a neural rendering layer, learning physically interpretable parameters of object sound models during an end-to-end learning task from real-world dataset. We evaluated its performance against state of the art audio learning baselines with both a human study and distance metrics, and found our model to be more expressive and resource efficient than alternatives for mod- eling real-world impact sounds. Finally, we showed how a robot can use our model to learn sound models of everyday objects in its environment by striking them in a noisy environment with every- day tools. The robot used our model to differentiate between the sounds of the tools it used and the objects it struck with them and could classify the materials of the objects it strikes from sound 140% more accurately than from the original, unseparated audio or from generic audio source separation baselines. The knowledge robots can use our model to ascertain about objects, materials, and contact forces will be valuable to future applications which use our approach in robot decision making and policy learning. As future work, we aim to expand the scope of our model to other contact events, such as rubbing and scratching, or other object acoustic events, such as bouncing on different surfaces or fracturing. We also intend to use our contributions as a foundation for models and applications which use other multimodal inputs, such as vision and tactile data.
8

---

Acknowledgments
We thank Christopher Atkeson, Ante Qu, Oliver Kroemer, Jacky Liang, and Leonid Keselman for valuable discussions, Kevin Zhang, Michael A. Lin, Hojung Choi, Brian Roberts, and Toki Migimatsu for assistance in setting up robotic experiments, and Matt Wright for assisting with audio recordings. This work is in part supported by the Toyota Research Institute (TRI), NSF CCRI #2 20095, Amazon Research Award (ARA), Autodesk, IBM, and the Stanford Institute for (luman-CCeteett AI (HAI).
N. Heravi was supported by the NSF Graduate Research Fellowship.
References
[1] Asmr bakery. URL
[2] J. Sinapov, C. Schenck, K. Staley, V. Sukhoy, and A. Stoytchev. Grounding semantic categories in behavioral interactions: Experiments with 100 objects. Robotics and Autonomous Systems, 62(5):632 (155)662-665,,, 2014. ISSN 0921-8890. bttpss//wwar sciencedirect. cww/sciennniieeccccocecccceieeoe /ar/aariiccclle//ee cial Issue Semantic Perception, Mapping and Exploration.
URL
Spe-
[3] J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. In 2009 JEEE International Conference on Robotics and Automation, ICRA 2009, Kobe, Japan, May 12-17, 2009, pages 2518-2524 IEEE, 2009. doi: 109/ROBOT. 2009.5152802. URL https: httptt:://doooooo /ttp:///ooo//////////////.. 10. 1101111000000 2009 5152802.
[4] D. Gandhi, A. Gupta, and I., Pinto. Swoosh! rattle! thump!-actions that sound. arXiv preprint arXiv:2007.0I
2020.
[5] J. Christie and N. Kottege. Acoustics based terrain classification for legged robots. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3596-3603, 2016. doi: 10.1 1.0.1109/16 10///CA..201....... 207774543..
[6] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. Proceedings of Machine Learning Research, 87, 2018.
[7] H. Liang, S. Li, X. Ma. N. Hendrich, T. Gerkmann, Sun, and J. Zhang. Making sense of audio vibration for liquid height estimation in robotic pouring, In IEEE International Conference on Intelligent Robots and Systems (IROS), 2019.
[8] C. Matl, Y. S. Narang, D. Fox, R. Bajesy, and Ramos. Stressd: Sim-to-real from sound for stochastic dynamics. CoRR, abs/2011.00 103136, 2020. URL https: org/ htt///bs/200000 /0111. 03136.
[9] N. Sainath and C. Parada. Convolutional neural networks for small-footprint keyword spot- ting. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
[10] H. Lee, P. Pham, Y. Largman, and A. Y. Ng. Unsupervised feature learning for audio classifi- cation using convolutional deep belief networks. In Advances in neural information processing systems, pages 1096-1 1904, 2009.
[11] Tulsiani, T. Zhou, A. A. Efros, and Malik. Multi-view supervision for single-view recon- struction via differentiable ray consistency, 2017.
[12] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis. Learning to estimate 3d human pose and shape from single color image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 459-468, 2018.
[13) M. Bao, M. Cong, S. Grabli, and R. Fedkiw. High-quality face capture using anatomical muscles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10802-1081 1,10001111111011111111 2019.
[14] A. Palazzi, L. Bergamini, S. Calderara, and R. Cucchiara. End-to-end 6-dof object pose timation through differentiable rasterization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0-0, 2018.

---

[15] J. Engel, L. Hantrakul, C. Gu, and A. Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint Stti:20001.04443.. 2020.
[16) R. Prenger, R. Valle, and B. Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEFE, 2019.
[17] A. V. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1rXvv:1609.04444 2016.
[18) K. K. d. Doel, D. L.. J.ng J. Lloyd, J. I. Richmond, and H. Yau. Scanning physical interaction behavior of 3d objects. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 87-96. 2001.
[19] K. Van Den Doel, P. G. Kry, and D. K. Pai. Foleyautomatic: physically-based sound effects for interactive simulation and animation. In Proceedings of 28th anmual conference on Computer graphics and interactive techniques, pages 537-544, 2001.
[20] M. Rau, O. Das, and E. Sanfield-Dafilov Improved carillon synthesis. in 22nd Int. Conf. Digital Audio Effects (DAFx-19) Birmingham, England, Sep. 2-6, 2019.
[21] Z. Ren, H. Yeh, and M. C. Lin. Example-gue guided physically based modal sound synthesis. ACM Transactions Graphics (TOG), 32(1):1-16. 2013.
[22] Z. Zhang, Q. Li, Z. Huang. J. Wu, J. Tenenbaum, and B. Freeman. Shape and material from sound. In Advances in Neural Information Processing Systems, pages 1278-1288, 2017.
[23] J. Traer, M. Cusimano, and J. H. McDermott. A perceptually inspired generative model of rigid- body contact sounds. In The 22nd International Conference Digital Audio Effects (DAFx-19), 2019.
[24] Engel, R. Swavely, L. H. Hantrakul, A. Roberts, and C. Hawthorne, Self-supervised pitch detection by inverse audio synthesis. 2020.
[25] N. Chadwick, C. Zheng, and D. L. James. Precomputed acceleration noise for improved rigid-body sound. ACM Transactions on Graphics (TOG), 31(4):1-9, 2012.
[26] Reed. Energy losses due elastic wave propagation during an elastic impact. Journal of Physics D: Applied Physics, 18(12):2329, 1985.
[27] D. L. James. Real-time rigid-body dynamics with acceleration noise.
http:
/ //graphics. stanf edu/ RBD_ AaDDAAcclNoisee .df. 2021.
[28] K. Johnson and K. L. Johnson. Contact mechanics. Cambridge university press, 1987.
[29] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint 192 2014.
[30] X. Serra and.J. J. Smith. Spectral modeling synthesis: A sound analysis/snnthesis system based on deterministic plus stochastic decomposition. Computer Music Journal, 14(4):12-24, 1990.
[31] P. Manocha, Z. Jin, R. Zhang, and A. Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity, 2021.
[32] P. Morgado. N. Vasconcelos, T. Langlois, and O. Wang. Self-supervised generation of spatial audio for 360 video. NeurlPS, 2018.
[33] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwal- dar, N. Haber, M. Sano, et al. Threedworld: platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.004954 2020.
10

---

[34] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, Straub. J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied Al Research. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2019.
[35] R. Gao, R. Feris, and K. Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018.
[36] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick. J. McDermott, and A. Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018.
[37] A. Ephrat, Mosseri, O. Lang, Dekel, Wilson, A. Hassidim, W. T. Freeman, and M. Ru- binstein. Looking to listen at the cocktail party: A speaker- -inker--ddeeendent audio-visua model for speech separation. arXiv preprint arXiv: 1804. 044.3369,, 2018.
[38] M. Spiertz and V. Gnann. Source-filter based clustering for monaural blind source separation. In Proceedings of the 12th International Conference on Digital Audio Effects, volume 4, 2009.
[39] Y. Tian, C. Xu, and D. Li. Deep audio prior. arXiv preprint arXiv: 19Iiv 10292,
[40] Z. Zhang, J. Wu, Q. Z. Huang. J. Traer, J. H. McDermott, J. B. Tenenbaum, and W. T. Free- man. Generative modeling of audible shapes for object perception. In The IEEE International Conference on Computer Vision 2017.
[41] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, A. Ragni, V. Valtchev, P. Woodland, and C. Zhang. The HTK Book (version 3.5a). 12 2015.
[42] B. Moore and B. R. Glasberg. A revision of zwicker's loudness model. Acta Acustica united with Acustica, 82(2):335 345, 1996.
[43] J. M. Ide. Comparison of statically and dynamically determined young's modulus of rocks. Proceedings of the National Academy of Sciences of the United States of America, 22(2):81, 1936.
[44] E. J. Chen, J. Novakofski, W. K. Jenkins, and W. D. O'Bren. Young's modulus mcasurements of soft tissues with application to elasticity imaging. IEEE Transactions ultrasonics, ferro- electrics, and frequency control, 1311)1191-194, 1996.
[45] J. Wang, W. Li, C. Lan, and P. Wei. Effective determination of young's modulus and poisson's 's ratio of metal using piezoelectric ring and electromechanical impedance technique: A proof-of- concept study. Sensors and Actuators A: Physical, 319:1 12561, 2021.
[46] B. Frank, R. Schmedding, C. Stachniss, M. Teschner, and W. Burgard. Learning the elasticity parameters of deformable objects with a manipulation robot. In 2010 IEEE/RS International Conference on Intelligent Robots and Systems, pages 1877-1883, IEBE, 2010.
[47] Boonvisut and M. C, Çavuşoğlu. Estimation soft tissue mechanical parameters from robotic manipulation data. IEEE/ASME Transactions on Mechatronics, 18(5): 1122-660216111111 2012.
[48] A. Davis. K. L. Bouman, G. Chen, M. Rubinstein, F. Durand, and W. T. Freeman. Visual vibrometry: Estimating material properties from small motion in video. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 5335-5343, 2015.
[49] D. James, J. Barbič, and D. K. Pai. Precomputed acoustic transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Trans. Graph., 25(3): 987-995, July 2006. ISSN 0730-0301. doi: 141911.1141983 URL StRlss/tt:///dt //doi.
org/10. 1 1145 11119111111111111 1141983.
[50] W. v. Schaik, M. Grooten, T. Wernaart, and C. v. d. Geld. High accuracy acoustic relative humidity measurement induct flow with air. Sensors, 1418):7421-743,, 2010.
11

---

[51] C. Z.Zenngaa and D. James. Toward high-quality modal contact sound. InACM SIGGRAPH 2011 papers, pages 1-12. 2011.
[52] V. Valimaki, J. D. Parker, L. Savioja, J. O. Smith, and S. Abel. Fifty years of artificial rever- beration. IEEE Transactions Audio, Speech, and Language Processing, 20(5): 20(5): 122--148, 2012.
[53] J. S. Abel, S. Coffin, and K. Spratt. A modal architecture for artificial reverberation with ap- plication to room acoustics modeling. In Audio Engineering Society Convention 137. Audio Engineering Society, 2014.
12


Reviews:
[{"review_id": "MoDMi9xQAH", "reviewer": "Reviewer_8sMD", "summary": "The paper introduces a new, differentiable audio model that can generate the sound of one or more rigid objects being struck. The authors show that by differentiating through the acoustic model, they can estimate material properties that allow them to identify objects.\n\nThe math for the audio signal generation is slightly out of my league but otherwise, I found the paper great. With the rise of differentiable physics engines, this was a matter of time.\nI couldn't find any major weakness that would prevent me from recommending the work.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Well-written, clear and detailed.\n- Experiments are great, both the in-the-wild ASMR data, human preference ranking, and the Franka Panda hitting the objects. Lots of effort and very illustrative of the strengths of your model.\n- Selection of baselines is good as far as I can tell. I'm no expert in audio generation but I liked the contrast of model-based and model-free methods.\n- Appendix is thorough with experimental details.\n\n### Weaknesses\n\n1. I can't really judge the quality of the audio generator formulas. Maybe some better reviewer can elucidate if this makes sense. However, I would have appreciated code or at least a jupyter notebook showing a little demo of the generator in the supplementary material.\n2. I've worked on a differentiable physics engine before and there it was the case that the system was very sensitive to the random initialization. You seemed to glance over that topic in a single sentence (line 205), without specifying the initialization ranges. Could you discuss a bit, please, if you found any issues here, e.g. if the model is diverging if a parameter is exceeding a certain distance from the GT, or if there are local minima that the model can get stuck in?\n3. I'd appreciate if you could write the optimization strategy in the robotic setup (sec 4.3-Optimization Setup) into a pseudocode block for clarity. That part wasn't 100% clear to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I enjoyed the paper and couldn't find any major weaknesses. But I remain open to others having issues with the math or lack of reproducibility.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No issues beyond the numbered weaknesses listed above.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "D8BA0bZlL84", "reviewer": "Reviewer_7DAN", "summary": "This paper proposes a realization of the physical processes that generate \"impact sound\" (i.e., the sound produced by an object striking another) into a differentiable programming framework. Doing so enables computing accurate, and---importantly---interpretable solutions to inverse problems, such as estimating physical properties (contact forces), and of learning models of impact sounds from real-world audio. Results are also presented on a real robot, on a sound source seperation task.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n\n**S1** Significant advance: This paper presents a significant advance in perception of impact sounds; and weaves together ideas from multiple disparate communities (ML, graphics, audio processing, robotics). Synthesizing realistic impact sounds is a yet-unsolved problem in the computer graphics and audio processing communities. Inverse problems like estimating physical parameters of contact from impact audio signals are currently tackled solely by \"model-free\" learning; these solution schemes lack physical interpretability. These are often compounded by the inherent noise in real-world audio recordings, and the combinatorially large sets of possible material interactions. Differentiable impact sound synthesis (DiffImpact) addresses the aforementioned problems in a single framework, by leveraging differentiable programming and gradient descent.\n\n**S2** Novelty / Topicality / Relevance: This paper adopts an emerging trend of baking in inductive biases about physical processes into a parameterized learning modules; to improve physical interpretability and dramatically reduce sample complexity. Differentiable programming has seen success with inverse problems in vision, graphics, and optimization (differentiable rendering, simulation, and optimization). To the best of my knowledge, this paper is the first to apply these ideas to impact sound synthesis.\n\n**S3** Breadth of experiments: I liked the generality of the framework; particularly the demonstration on real-world videos. The audio samples produced by DiffImpact (from the supplementary material) are of superior quality to other model-free baselines, demonstrating the need for inductive bias (differentiable impact sound modeling). As opposed to a pure system-identification approach (which is the common mode of deploying differentiable physics priors), DiffImpact trains an autoencoder for its impact sound modeling (Sec. 4.2), improving the quality of output audio.\n\n**S4** Clarity / exposition: Overall, this paper was a joy to read -- very well-written and adequately positioned w.r.t. relevant prior work.\n\n\nWeaknesses\n==========\n\nI have a few questions that I hope to see discussed during the review cycle.\n\n**W1** Justification of assumed quantities: From the paper and the supplementary materials, it is unclear as to what quantities (and if applicable, what priors over these quantities) are assumed given. For the experiments in 4.1, is the microphone assumed to be static (and is a single microphone assumed)? More importantly, are all quantities fully observable? I would presume some pairs of quantities may not be co-observable (and therefore may not be jointly estimated; E.g., the material properties and contact force parameters). It would help to add a note / table to this effect.\n\nAn addendum to the above discussion point: is it technically feasible to retrieve physically grounded contact parameters (not just proportional quantities) by assuming some of the quantities are known (or a strong prior over their absolute values is provided?\n\n\n**W2** Clarification on environmental audio: From the description of the room impulse response (Sec. 3.4), it seems like what's considered environment noise is actually reverberations of the impact sound *without regards to the geometry of the environment*? That is, the model does not account for the precise manner in which sound reflects off of various surfaces in the environment, but rather lumps all of these terms into a single \"ambient noise\"? If so, this could be made more explicit in the paper.\n\n\n**W3** Granularity for the classification task: Arguably, among all tasks presented in this paper, the source separation task (Sec. 4.3) is one that has seen the most interest from the ML community. Classification accuracies for source separation rely on the set of object materials chosen. The four chosen materials are quite distinctive (and perhaps a small sample size). It would therefore make for a more interesting experiment to compare the absolute values of estimated material properties (as opposed to computing a classification accuracy).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, this paper presents a novel idea that constitutes a significant advance in processing impact audio from microphones. This paper has the potential to spur several interesting follow-up directions. This paper is quite interdisciplinary as well, which I see as a promising sign.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I have a few minor, yet relevant, issues (W1 through W3) that I would like to discuss in the revision phase.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "8x4OkfQ6__R", "reviewer": "Reviewer_FKcr", "summary": "The paper presents a network architecture (DiffImpact) and analytic modeling scheme for differentiable rendering of impact sounds for rigid objects. Analytic modeling choices such as impulse force and impulse response are taken from prior literature and adapted accordingly, such as taking a Gaussian approximation to impulse force and allowing for multiple contacts. Environment noise and reverb from the environment are also accounted for. The model is trained with ground truth spectrogram as input and there are separate generator modules for impact force profile, impulse response, environment noise, and environment reverb. The outputs of these modules are combined to produce a prediction of a final waveform, where an L1 loss supervises the learning against the ground truth spectrogram. Downstream tasks include reproducing waveforms from ASMR videos, sound separation for a robot hitting an object with a tool (i.e. generating separate waveforms for the impacted object and the tool the robot hit object with), and classifying object material. Both quantitative and qualitative results are shown, where the latter used a Mechanical Turk user study to compare preferences of the generated waveforms of DiffImpact to other baselines including a state-of-art speech generator. Qualitative results show users preferred DiffImpact over the baselines for all objects except a deformable object, which violates the rigid body assumptions of the paper. On the object material prediction task, DiffImpact (separating tool and object impact sounds) significantly improves classification accuracy over using the original waveforms.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The experimental results are very impressive, particularly separating the waveforms of the tool and object being impacted. This can be extremely useful for robot manipulation when the robot has prior knowledge about the tool (e.g. it’s just tapping with its own finger/end-effector).\n+ The generated waveforms for the ASMR videos sound very realistic, effectively noise-filtered versions of the ground truth sounds for several of the objects. It’s also intuitive that the method fails on the deformable object given the method’s rigid body assumption.\n+ The downstream task of material classification is a nice showcase of the method.\n+ The paper is clearly written. I am not an expert in audio modeling, yet I could follow the approach very well and understand the technical details provided in the appendix. The paper provides sufficient high-level information in the main text and sufficient technical details in the appendix.\nWeaknesses:\n- It would be beneficial to show robot experiments that demonstrate the approach can benefit robot policy learning and decision-making in some way, as opposed to only using the robot to collect data. For example, a task where success is dependent on using an object that is composed of a particular material (e.g. a metallic object that is heavier than the others), and show DiffImpact enables finding and using that object better than other baselines.\n- The model input is a spectrogram which depending on the task for the robot may have limited utility. An architecture that takes in multimodal inputs (e.g. the video, audio, force sensor readings, joint state, etc.) would provide more downstream flexibility and allow for cross-modal predictions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the robot experiments do not directly show the benefits of the approach for performing robot tasks, the prospective benefits of utilizing the method for downstream robot manipulation tasks is promising. Audio is a very under-utilized modality for real-robot manipulation, in part because of the difficulty in acquiring a useful signal that isn’t polluted by background noise and distractor sounds. This approach could feasibly allow the robot to reason in a targeted way about the audio signals generated only from its intentional interactions with object and filter out the unwanted distractors. Utilizing this method with other robot sensory modalities and connecting the features with the robot’s action space seems like a truly promising extension to this work, and would be tractable to pursue given it is a neural network that can be integrated to other architectures. I therefore think this approach has promise to inspire further research and extensions that could have a big impact on robot manipulation research.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The use of a robot is limited, primarily just for automated data collection. It’s true material classification is a downstream task that is useful in robotics, but it would be more compelling to show a robot can perform a task in an improved way using the proposed method. I think it is sufficient that the proposed approach is promising for further research that can incorporate the approach into policy learning schemes, but perhaps more can be said in the paper about potential applications in autonomous robot learning."}}]

Author rebuttal:

Meta-review: All three reviewers are very positive about this paper. Whilst the paper does not explicitly involve robotics experiments, the reviewers agree that there is good future potential in this direction, and furthermore, this is an interesting area of robotics which is rarely addressed by the community. The main weaknesses come from queries and requests for clarifications; please address these in the rebuttal to maintain these positive reviews.

-------

Following the reviews, the authors have addressed a number of queries by the reviewers, and all the reviewers have maintained their recommendation of acceptance. This paper presents an interesting solution for sound synthesis, with some encouraging results on material classification. Whilst the use of this method for actually controlling a robot is not yet clear, it may lead to interesting discussions in the community about how we should be using sound information for robotics, which still remains a relatively unexplored field.
Decision: Accept (Oral)