Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This very nice paper tackles the integration of of domain knowledge in signal processing within neural nets; the approach is illustrated in the domain of music. 

The proof of concept (audio supplementary material) is very convincing. 

The argument is that a "natural" latent space for audio is the spectro-temporal domain. Approaches working purely in the waveform, or in the frequency domains must handle the phase issues. Approaches can learn to handle these issues, at the expense of more data. 
A key difficulty is that the L_2 loss does not match the perception. The authors present a perceptual loss that addresses the point (Eq. 4 - clarify the difference w.r.t. Wang et al.), . A natural question thus is whether applying this loss on e.g. Wavenet, Sample RNN or Wave RNN would solve the problem.

In short, the contribution is in designing a latent space that accounts for independent components of the music signal (pitch; loudness; reverberation and/or noise), using existing components (oscillators, envelopes and filters), and making them amenable to end-to-end optimization (noting that loudness can be extracted deterministically).

I understand that the auto-encoder is enriched with a FIR filter at its core: the input is mapped into the time-frequency domain; convolved with the output of the neural net H_l, and the result is recovered from the time-frequency domain. 
Explain "frames x_l to match the impulse responses h_l".

Care (domain knowledge and trials and errors, I suppose) is exercized in the conversion and recovery (shape and size of the window) to remove undesired effects. 

Overall, the approach works in two modes: one where the fundamental frequency is extracted, one where it is learned. I miss this comparison in the audio material, could you tell where to look / hear ? 


Questions:
* NN operate at a slower frame rate (sect. 3.2): how much slower ? How sensitive to this parameter ?

Detail
* were, end p. 5. A word missing ?  
* are useful --> is useful ? 
* could produced, p.6 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a model for audio generation/synthesis  where the model is trained to output time-varying parameters of a vocoder/synthesiser rather than directly outputting audio samples/spectrogram frames. The model is trained by mining an L1 loss between the synthesised audio and the real training audio. Overall, I found the paper to be well written, I found the online supplementary material helpful in getting an intuition for the quality of audio samples generated and to understand the various parts of the proposed architecture. I also found the figures to be extremely  informative, especially figure 2. Although, I think the title of the paper could be updated to something more specific like Differentiable Vocoders or something similar, since the description and experiments very specifically deal with audio synthesis with vocoders, even though the components might be general DSP components. I think the paper presents a reasonable alternative to current autoregressive audio generation methods and should be accepted for publication. 

Minor Comments

1. The reference provided for RNNs, Sutskever et. al. 2014, should be supplemented with older references from the 80s when RNNs were first trained with backdrop through time. 
2. “The bias of the natural world is to vibrate.” I am not sure what exactly is meant here. Is it really a “bias” if every object in the universe vibrates? I think the term bias has been overloaded in the paper and is confused with structural/inductive priors. Bias as used in the paper, seems to have several means. It would help the description if the authors cleared up this confusing use of the term. 
3. In Section 1.3, the authors claim that one of the strengths of the proposed method is that the models are “relatively small”, however all discussion of the sizes is relegated to the appendix. It would be helpful to the reader if some model complexity comparisons are presented in the results section and a high level summary is presented at the end of Section 1. 
4. In Section 3, some additional high-level description for the Harmonic plus Noise model (Serra & Smith, 1990) should be provided to motivate the discussion and experiments in the rest of the paper. 
5. Section 4.2 should provide some description of the parameters counts for each of the components considered and how this compares with existing auto-regressive generation algorithms. 

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper develops a framework for for audio generation using oscillators with differentiable neural network type learning. They showcase the usefulness and effectiveness of the approach with several examples such as timbre transfer, dereverberation, changing the room impulse response, pitch extrapolation and so on. I can imagine the proposed learnable oscillator based autoencoders in a variety of applications. 

I think that this suggested software library can be useful for a wide range of audio researchers, and I commend the authors for this contribution. It is very nice to see an example of research where we make use of our physical understanding of the sound medium rather than blindly throwing a neural network at the problem. 

I have one important question though: how susceptible do you think the system is robust with respect to f0 and loudness encoders? Have you experimented with situations where the f0 and the loudness encoders might fail (such as more non-periodic and noisy signals)? 

review_title: Official Blind Review #1

Meta-review: This paper proposes a novel differentiable digital signal processing in audio synthesis. The application is novel and interesting. All the reivewers agree to accept it. The authors are encouraged to consider the reviewer's suggestions to revise the paper.
Decision: Accept (Spotlight)