Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a realization of the physical processes that generate "impact sound" (i.e., the sound produced by an object striking another) into a differentiable programming framework. Doing so enables computing accurate, and---importantly---interpretable solutions to inverse problems, such as estimating physical properties (contact forces), and of learning models of impact sounds from real-world audio. Results are also presented on a real robot, on a sound source seperation task.
strengths_and_weaknesses: Strengths
=========

**S1** Significant advance: This paper presents a significant advance in perception of impact sounds; and weaves together ideas from multiple disparate communities (ML, graphics, audio processing, robotics). Synthesizing realistic impact sounds is a yet-unsolved problem in the computer graphics and audio processing communities. Inverse problems like estimating physical parameters of contact from impact audio signals are currently tackled solely by "model-free" learning; these solution schemes lack physical interpretability. These are often compounded by the inherent noise in real-world audio recordings, and the combinatorially large sets of possible material interactions. Differentiable impact sound synthesis (DiffImpact) addresses the aforementioned problems in a single framework, by leveraging differentiable programming and gradient descent.

**S2** Novelty / Topicality / Relevance: This paper adopts an emerging trend of baking in inductive biases about physical processes into a parameterized learning modules; to improve physical interpretability and dramatically reduce sample complexity. Differentiable programming has seen success with inverse problems in vision, graphics, and optimization (differentiable rendering, simulation, and optimization). To the best of my knowledge, this paper is the first to apply these ideas to impact sound synthesis.

**S3** Breadth of experiments: I liked the generality of the framework; particularly the demonstration on real-world videos. The audio samples produced by DiffImpact (from the supplementary material) are of superior quality to other model-free baselines, demonstrating the need for inductive bias (differentiable impact sound modeling). As opposed to a pure system-identification approach (which is the common mode of deploying differentiable physics priors), DiffImpact trains an autoencoder for its impact sound modeling (Sec. 4.2), improving the quality of output audio.

**S4** Clarity / exposition: Overall, this paper was a joy to read -- very well-written and adequately positioned w.r.t. relevant prior work.


Weaknesses
==========

I have a few questions that I hope to see discussed during the review cycle.

**W1** Justification of assumed quantities: From the paper and the supplementary materials, it is unclear as to what quantities (and if applicable, what priors over these quantities) are assumed given. For the experiments in 4.1, is the microphone assumed to be static (and is a single microphone assumed)? More importantly, are all quantities fully observable? I would presume some pairs of quantities may not be co-observable (and therefore may not be jointly estimated; E.g., the material properties and contact force parameters). It would help to add a note / table to this effect.

An addendum to the above discussion point: is it technically feasible to retrieve physically grounded contact parameters (not just proportional quantities) by assuming some of the quantities are known (or a strong prior over their absolute values is provided?


**W2** Clarification on environmental audio: From the description of the room impulse response (Sec. 3.4), it seems like what's considered environment noise is actually reverberations of the impact sound *without regards to the geometry of the environment*? That is, the model does not account for the precise manner in which sound reflects off of various surfaces in the environment, but rather lumps all of these terms into a single "ambient noise"? If so, this could be made more explicit in the paper.


**W3** Granularity for the classification task: Arguably, among all tasks presented in this paper, the source separation task (Sec. 4.3) is one that has seen the most interest from the ML community. Classification accuracies for source separation rely on the set of object materials chosen. The four chosen materials are quite distinctive (and perhaps a small sample size). It would therefore make for a more interesting experiment to compare the absolute values of estimated material properties (as opposed to computing a classification accuracy).
summary_of_recommendation: In summary, this paper presents a novel idea that constitutes a significant advance in processing impact audio from microphones. This paper has the potential to spur several interesting follow-up directions. This paper is quite interdisciplinary as well, which I see as a promising sign.
issues: I have a few minor, yet relevant, issues (W1 through W3) that I would like to discuss in the revision phase.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper introduces a new, differentiable audio model that can generate the sound of one or more rigid objects being struck. The authors show that by differentiating through the acoustic model, they can estimate material properties that allow them to identify objects.

The math for the audio signal generation is slightly out of my league but otherwise, I found the paper great. With the rise of differentiable physics engines, this was a matter of time.
I couldn't find any major weakness that would prevent me from recommending the work.
strengths_and_weaknesses: ### Strengths

- Well-written, clear and detailed.
- Experiments are great, both the in-the-wild ASMR data, human preference ranking, and the Franka Panda hitting the objects. Lots of effort and very illustrative of the strengths of your model.
- Selection of baselines is good as far as I can tell. I'm no expert in audio generation but I liked the contrast of model-based and model-free methods.
- Appendix is thorough with experimental details.

### Weaknesses

1. I can't really judge the quality of the audio generator formulas. Maybe some better reviewer can elucidate if this makes sense. However, I would have appreciated code or at least a jupyter notebook showing a little demo of the generator in the supplementary material.
2. I've worked on a differentiable physics engine before and there it was the case that the system was very sensitive to the random initialization. You seemed to glance over that topic in a single sentence (line 205), without specifying the initialization ranges. Could you discuss a bit, please, if you found any issues here, e.g. if the model is diverging if a parameter is exceeding a certain distance from the GT, or if there are local minima that the model can get stuck in?
3. I'd appreciate if you could write the optimization strategy in the robotic setup (sec 4.3-Optimization Setup) into a pseudocode block for clarity. That part wasn't 100% clear to me.
summary_of_recommendation: I enjoyed the paper and couldn't find any major weaknesses. But I remain open to others having issues with the math or lack of reproducibility.
issues: No issues beyond the numbered weaknesses listed above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a network architecture (DiffImpact) and analytic modeling scheme for differentiable rendering of impact sounds for rigid objects. Analytic modeling choices such as impulse force and impulse response are taken from prior literature and adapted accordingly, such as taking a Gaussian approximation to impulse force and allowing for multiple contacts. Environment noise and reverb from the environment are also accounted for. The model is trained with ground truth spectrogram as input and there are separate generator modules for impact force profile, impulse response, environment noise, and environment reverb. The outputs of these modules are combined to produce a prediction of a final waveform, where an L1 loss supervises the learning against the ground truth spectrogram. Downstream tasks include reproducing waveforms from ASMR videos, sound separation for a robot hitting an object with a tool (i.e. generating separate waveforms for the impacted object and the tool the robot hit object with), and classifying object material. Both quantitative and qualitative results are shown, where the latter used a Mechanical Turk user study to compare preferences of the generated waveforms of DiffImpact to other baselines including a state-of-art speech generator. Qualitative results show users preferred DiffImpact over the baselines for all objects except a deformable object, which violates the rigid body assumptions of the paper. On the object material prediction task, DiffImpact (separating tool and object impact sounds) significantly improves classification accuracy over using the original waveforms.

strengths_and_weaknesses: Strengths:
+ The experimental results are very impressive, particularly separating the waveforms of the tool and object being impacted. This can be extremely useful for robot manipulation when the robot has prior knowledge about the tool (e.g. it’s just tapping with its own finger/end-effector).
+ The generated waveforms for the ASMR videos sound very realistic, effectively noise-filtered versions of the ground truth sounds for several of the objects. It’s also intuitive that the method fails on the deformable object given the method’s rigid body assumption.
+ The downstream task of material classification is a nice showcase of the method.
+ The paper is clearly written. I am not an expert in audio modeling, yet I could follow the approach very well and understand the technical details provided in the appendix. The paper provides sufficient high-level information in the main text and sufficient technical details in the appendix.
Weaknesses:
- It would be beneficial to show robot experiments that demonstrate the approach can benefit robot policy learning and decision-making in some way, as opposed to only using the robot to collect data. For example, a task where success is dependent on using an object that is composed of a particular material (e.g. a metallic object that is heavier than the others), and show DiffImpact enables finding and using that object better than other baselines.
- The model input is a spectrogram which depending on the task for the robot may have limited utility. An architecture that takes in multimodal inputs (e.g. the video, audio, force sensor readings, joint state, etc.) would provide more downstream flexibility and allow for cross-modal predictions.

summary_of_recommendation: While the robot experiments do not directly show the benefits of the approach for performing robot tasks, the prospective benefits of utilizing the method for downstream robot manipulation tasks is promising. Audio is a very under-utilized modality for real-robot manipulation, in part because of the difficulty in acquiring a useful signal that isn’t polluted by background noise and distractor sounds. This approach could feasibly allow the robot to reason in a targeted way about the audio signals generated only from its intentional interactions with object and filter out the unwanted distractors. Utilizing this method with other robot sensory modalities and connecting the features with the robot’s action space seems like a truly promising extension to this work, and would be tractable to pursue given it is a neural network that can be integrated to other architectures. I therefore think this approach has promise to inspire further research and extensions that could have a big impact on robot manipulation research.
issues: The use of a robot is limited, primarily just for automated data collection. It’s true material classification is a downstream task that is useful in robotics, but it would be more compelling to show a robot can perform a task in an improved way using the proposed method. I think it is sufficient that the proposed approach is promising for further research that can incorporate the approach into policy learning schemes, but perhaps more can be said in the paper about potential applications in autonomous robot learning.
review_title: null

Meta-review: All three reviewers are very positive about this paper. Whilst the paper does not explicitly involve robotics experiments, the reviewers agree that there is good future potential in this direction, and furthermore, this is an interesting area of robotics which is rarely addressed by the community. The main weaknesses come from queries and requests for clarifications; please address these in the rebuttal to maintain these positive reviews.

-------

Following the reviews, the authors have addressed a number of queries by the reviewers, and all the reviewers have maintained their recommendation of acceptance. This paper presents an interesting solution for sound synthesis, with some encouraging results on material classification. Whilst the use of this method for actually controlling a robot is not yet clear, it may lead to interesting discussions in the community about how we should be using sound information for robotics, which still remains a relatively unexplored field.
Decision: Accept (Oral)