DDSP: Differentiable Digital Signal Processing
Authors: Jesse Engel
Lamtharn (Hanoi) Hantrakul
Chenjie Gu
Adam Roberts
Venue:  (2020)
Keywords: dsp
audio
music
nsynth
wavenet
wavernn
vocoder
synthesizer
sound
signal
processing
tensorflow
autoencoder
disentanglement

Abstract: Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library will is available at https://github.com/magenta/ddsp and we encourage further contributions from the community and domain experts.


Published as a conference paper at ICLR 2020
DDSP:
DIFFERENTIABLE DIGITAL SIGNAL PROCESSING
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, & Adam Roberts Google Research, Brain Team Mountain View, CA 94043, USA {jesseengel, hanoih, gei, adarob)
ABSTRACT
Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do utilize existing knowledge of how sound is generated and perceived. A third approach (oocoders/synthesiizess)) successfully incorpo- rates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern autto-iiffrrentaation-aeee introduce the Differentiable Digital Signal Processing (DDSP) library, which en- ables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation with- out the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training. blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between dis- parate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available and we welcome further contributions from the community and domain experts.
machine learning methods. In this paper, we
INTRODUCTION
Neural networks are universal function approximators in the asymptotic limit (Hornik al. but their practical success is largely due to the use of strong structural priors such as convolution (Le- Cun et al. 1989), recurrence (Sutskever et 2014 Williams & Zipser 1990 Werbos 1990). and sell-attention (Vaswani et al. 2017). These architectural constraints promote generalization and data efficiency the extent that they align with the data domain. From this perspective, end-to-end learning relies on structural priors to scale, but the practitioner's 's toolbox is limited to functions that can be expressed differentiably. Here, we increase the size of that toolbox by introducing the Differ- entiable Digital Signal Processing (DDSP) library, which integrates interpretable signal processing elements into modern automatic differentiation software (TensorFlow). While this approach has broad applicability, we highlight its potential in this paper through exploring the example of audio synthesis.
1989)
Objects have natural tendency to periodically vibrate. Small shape displacements are usually restored with elastic forces that conserve energy (similar canonical mass on spring), leading to harmonic oscillation between kinetic and potential energy (Smith) 2010). Accordingly, human hearing has evolved to be highly sensitive phase-coherent oscillation, decomposing audio into spectrotemporal responses through the resonant properties of the basilar membrane and tonotopic
'Onliee Resources:
Code: htteps///ggttauu...cc//aaeetttet
Audio Examples: http:/pp::./ggoooo///eeeeectt/ee ddwp:ttaaaddpppdddeeeeeeeeeeeeee examp) les
Colab Demo: http:s///oo..////ggg////////pppp

---

Published as a conference paper at ICLR 2020
mappings into the auditory cortex (Moerel al. 2012 Chi al. 2005 Theunissen & Elie 2014). However, neural synthesis models often do not exploit this periodic structure for generation and perception.
1.1 CHALLENGES OF NEURAL AUDIO SYNTHESSS
As shown in Figure D most neural synthesis models generate waveforms directly in the time domain, from their corresponding Fourier coefficients in the frequency domain. While these representa- tions are general and can represent any waveform, they are not free from bias. This is because they often apply prior over generating audio with aligned wave packets rather than oscillations. For ample, strided convolution models--uche as SING (Defossez et al. 2018), MCNN (Arik et al. 2019), and WaveGAN (Donahue et al., 2019))-enerae generate waveforms directly with overlapping frames. Since audio oscillates at many frequencies, all with different periods from the fixed frame hop size, the model must precisely align waveforms between different frames and learn filters to cover all possible phase variations. This challenge is visualized on the left of Figure 1
Fourier-based models-such Tacotron (Wang et al.) 2017) and GANSynth (Engel et al. 2019 also suffer from the phase-alignment problem, as the Short-time Fourier Transform (STFT) T) representation over windowed wave packets. Additionally, they must contend with spectral leakage, where sinusoids at multiple neighboring frequencies and phases must be combined to represent a single sinusoid when Fourier basis frequencies do not perfectly match the audio. This effect can be seen in the middle diagram of Figure
Autoregressive waveform models-such WaveNet (Oord et al. 2016), SampleRNN (Mehri al. 2016), and WaveRNN (Kalchrrenner et al. -avoid these issues by generating the waveform single sample at time. They are not constrained by the bias over generating wave packets and can express arbitrary waveforms. However, they require larger and more data-hungry networks, as they do not take advantage of bias over oscillation (size comparisons can be found in Table B.6). Furthermore,
use of teacher-forcing during training leads to exposure bias during generation, where errors with feedback compound. It also makes them incompatible with perceptual losses such as spectral features (Defossez et al., (2018), pretrained models (Dosovitskiy & Brox 2016), and discriminators (Engel et al.) 2019). This adds further inefficiency to these models, waveform's 's shape does not perfectly correspond to perception. For example, the three waveforms on the right of Figure sound identical (a relative phase offset of the harmonies) but would present different losses to an autoregressive model.
Strided Convolution
Fourier Representation Spectral Leakage (Tacotron, GANSynth)
Autoregressive Waveform Perception
Phase Alignment (WaveGAN, SING)
(WaveNet, SampleHNN)
Figure 1: Challenges of neural audio synthesis. Full description provided in Section [.]
1.2 OSCILLATOR MODELS
Rather than predicting waveforms or Fourier coefficients, a third model class directly generates audio with oscillators. Known vocoders synthesizers, these models physically and percep- tally motivated and have a long history of research and applications (Beauchamp) (2077: Morise et al. 2016). These "analysis/syttheis"" models expert knowledge and hand-tuned heuristics to extract synthesis parameters (analysis) that are interpretable (loudness and frequencies) and can be used by the generative algorithm (synthesis).
Neural networks have been used previously to some success in modeling pre-extracted synthesis parameters Blaauu & Bonada 2017, Chandna al. 2019), but these models fall short of end- to-end learning. The analysis parameters must still be tuned by hand and gradients cannot flow through the synthesis procedure. As result, small errrr in parameters can lead to large emror in
2

---

Published as a conference paper at ICLR 2020
the audio that cannot propagate back to the network. Crucially, the realism of vocoders is limited by the expressivity of given analysis/synthesis pair.
1.3 CONTRIBUTIONS
In this paper, overcome the limitations outlined above by using the DDSP library implement fully differentiable synthesizers and audio effects. DDSP models combine the strengths of the above approaches, benefiting from the inductive bias of using oscillators, while retaining the expressive power neural networks and end-to-end training.
We demonstrate that models employing DDSP components are capable of generating high-fidelity audio without autoregressive or adversarial losses. Further, we show the interpretability and modu- larity of these models enable:
Independent control over pitch and loudness during synthesis.
Realistic extrapolation to pitches not seen during training.
Blind dereverberation of audio through seperate modelling of room acoustics.
Transfer of extracted room acoustics to new enviromments.
Timbre transfer between disparate sources, converting singing voice into violin.
Smaller network sizes than comparable neural synthesizers.
Audio samples for all examples and figures are provided in the online supplement We highly encourage readers to listen to the samples as part of reading the paper.
2 RELATED WORK
Vocoders. Vocoders come in several varieties. situeeeiller/suubraatvee models are inspired by the human vocal tract and dynamically filter a harmonically rich source signal (Flanagan (013). while sinusoidal/additive models generate sound the combination of set of time-varying sine waves (McAulay & Quatieri] 1986 |Serra & Smith] 1990). Additive models are strictly more expres- sive than subtractive models but have more parameters each sinusoid has its own time-varying loudness and frequency. This work builds a differentiable synthesizer off the Harmonic plus Noise model (Serra & Smith, [990, [eaacchamp 2007): an additive synthesizer combines sinusoids in harmonic (integer) ratios of a fundamental frequency alongside a time-varying filtered noise signal.
Synthesizers. A separate thread of research has tried to estimate parameters for commercial syn- thesizers using gradient-free methods (Huang et al.] [201]] Hoffman & Cook) 2006).. Synthesizer outputs modeled with variational autoencoder were recently used "world model" (Ha & Schmidhuber] 2018) to pass approximate gradients to controller during learning (Esling et al.) 2019). DDSP differs from black-box approaches modeling existing synthesizers; it is toolkit of differentiable DSP components for end-to-end learning.
Neural Source Filter (NSF). Perhaps closest to this work, promising speech synthesis results were recently achieved using a differentiable waveshaping synthesizer (Wang et al., 2019). The NSF can be seen specific DDSP model, that uses convolutional waveshaping of sinusoidal oscillator to create harmonic content, rather than additive synthesis explored in this work. Both works also generate audio in the time domain and impose multi-scale spectrograms losses in the frequency domain. A key contribution of this work is to highlight how these models are part of a common family of techniques and to release modular library that makes them accessible by leveraging automatic differentiation to easily mix and match components at a high level.
DDSP COMPONENTS
Many DSP operations can be expressed as functions modern automatic differentiation software. We express core components as feedforward functions, allowing efficient implementation parallel
https::///oot http://gww...//mmaaeaaa///////
3

---

Published as a conference paper at ICLR 2020
hardware such GPUs and TPUs, and generation of samples during training. These components include oscillators, envelopes, and filters (linear-timee time-varying cae-iiippuuuleuonneee LTV-FIR).
3.1 SPECTRAL MODELING SYNTHESIS
Here, example DDSP model, implement differentiable version of Spectral Modeling Synthesis (SMS) Serra & Smith (1990). This model generates sound by combining an additive synthesizer (adding together many sinusoids) with subtractive synthesizer (filtering white noise). We choose SMS because, despite being parametric, it is a highly expressive model of sound, and has found widespread adoption in tasks diverse as spectral morphing. time stretching, pitch shifting, source separation, transcription, and even as a general purpose audio codec in MPEG-4 (Tellman al. 1995 Klapuri et al. 2000 Purnhagen & Meine 2000).
As we only consider monophonic sources in these experiments, we use the Harmonic plus Noise model, that further constrains sinusoids to be integer multiples of fundamental fre- quency (Beauchamp) 2007). One of the reasons that SMS is more expressive than many other parametrie models because it has many more parameters. For example, in the 4 seconds of 16kHz audio in the datasets considered here, the synthesizer coefficients actually have ~2.5 times more dimensions than the audio waveform itself ((1 amplitude 100 harmonics + 65 noise band magnitudes) * 1000 timesteps = 165,000 dimensions, vs. 64,000 audio samples). This makes them amenable to control by neural network, it would be difficult realistically specify all these parameters by hand.
3.2 HARMONIC OSCILLATOR / ADDITIVE SYNTHESIZER
At the heart of the synthesis techniques explored in this paper the sinusoidal oscillator, bank of oscillators that outputs a signal (n1) over discrete time steps, n, can be expressed as:
x (n)
Σ (n(m) (n)(
(1)
where (n( is the time-varying amplitude of the k-th sinusoidal component and (x(n) is its in- stantaneous phase. The phase Φ& (n) is obtained by integrating the instantaneous frequency fn((n):
(n) 2m
Σ fa(m) + do,k:
(2)
m==
where Φ is the initial phase that can be randomized, fixed, or learned.
For harmonic oscillator, all the sinusoidal frequencies are harmonic (integer) multiples of funda- mental frequency, fo(n), i.e. [u(nn) kfo(n), Thus the output of the harmonic oscillator is entirely parameterized by the time-varying fundamental frequency fo(n) and harmonic amplitudes A₁ (n). To aid interpretablity we further factorize the harmonic amplitudes:
(n = (m))
into global amplitude A(r) that controls the loudness and normalized distribution over harmonics
c(n) that determines spectral variations, where Σ cx (n) = 1 and eı (n)) 2 O. We also constrain both amplitudes and harmonic distribution components to be positive through the use of a modified sigmoid nonlinearity as described in the appendix. Figure 6 provides a graphical example of the additive synthesizer. Audio is provided in our online supplement
3.3 ENVELOPES
The oscillator formulation above requires time-varying amplitudes and frequencies at the audio sam- ple rate, but our neural networks operate at slower frame rate. For instantaneous frequency upsam- pling, we found bilinear interpolation to be adequate. However, the amplitudes and harmonic distri- butions of the additive synthesizer required smoothing to prevent artifacts. We are able to achieve
We have implemented further components such wavetable synthesizers and non-sinusoidal oscillators, but focus here on components used in the experiments and leave the rest as future work.
4

---

Published as a conference paper at ICLR 2020
this with a smoothed amplitude envelope by adding overlapping Hamming windows the center of each frame and scaled by the amplitude. For these experiments we found a 4ms (64 timesteps) hop size and ms frame size (50% overlap) to be responsive to changes while removing artifacts.
3.4 FILTER DESIGN: FREQUENCY SAMPLING METHOD
Linear filter design is a comeestone of many DSP techniques. Standard convolutional layers are equivalent to linear time invariant finite impulse response (LTI-FIR) filters. However, to ensure interpretability and prevent phase distortion, we employ the frequency sampling method to convert network outputs into impulse responses linear-phase filters.
Here, we design a neural network to predict the frequency-domain transfer functions of a FIR filter for every output frame. In particular, the neural network outputs vector H, (and accordingly h = IDFF(HH)) for the I-th frame of the output. We interpret H₃ as the frequency-domate transfer function of the corresponding FIR filter. We therefore implement a time- varying FIR filter.
To apply the time-varying FIR filter to the input, we divide the audio into non-overlapping frames match the impulse responses hy. We then perform frame-wis wise convolution via multiplication of frames in the Fourier domain: Y ; where X; DFT(x:) (ax) and Y = DFT(y:) is the output. We recover the frame-wise filtered audio, yı = IDFT Y)). and then overlap-ade the resulting frames with the same hop size and rectangular window used to originally divide the input audio. The hop size given by dividing the audio into equally spaced frames for each frame of conditioning. For 64000 samples and 250 frames, this corresponds to hop size of 256.
In practice, we do function W on the network output to compute H,. The shape and size of the window can be decided independently control the time-frege frequeccy resolution trade-off the filter. In our experiments, we default to a Hann window of size 257. Without window, the resolution implicitly defaults to rectangular window which is not ideal for many cases. We take care to shift the IR to zero-pla zero-phase (symmetric) form before applying the window and revert to causal form before applying the filter.
the neural network output directly Hi Instead, we apply window
3.5 FILTERED NOISE SUBTRACTIVE SYNTHESIZER
Natural sounds contain both harmonic and stochastic components. The Harmonic plus Noise model captures this by combining the output of an additive synthesizer with stream of filtered noise (Serra & Smith 1990, [Beauchamp] 2007). We are able to realize a differentiable filtered noise synthesizer by simply applying the LTV-FIR filter from above stream of uniform noise Y, H, where N; is the DFT of uniform noise in domain [-1, 1].
3.6 REVERB: LONG IMPULSE RESPONSES
Room reverbation (reverb) is an essential characteristic of realistic audio, which is usually implicitly modeled by neural synthesis algorithms. In contrast, gain interpretability by explicitly factoriz- ing the room acoustics into a post-synthesis convolution step. A realistic room impulse response (IR) can be long several seconds, corresponding extremely long convolutional kernel sizes (~10-100k timesteps). Convolution via matrix multiplication scales as O(n²), which is intractable for such large kernel sizes. Instead, we implement reverb by explicitly performing convolution as multiplication in the frequency domain, which scales as O(n log rı) and does not bottleneck training.
4 EXPERIMENTS
For empirical verification of this approach, test two DDSP autoencoder variants-supervian supervised and unsupervised-on two different musical datasets: NSynth (Engel et al. (0077) and a collection of solo violin performances. The supervised DDSP autoencoder is conditioned on fundamental frequency (FO) and loudness features extracted from audio, while the unsupervised DDSP autoencoder learns FO jointly with the rest of the network.
5

---

Published as a conference paper at ICLR 2020
Harmonic Audio
FO
Encoder
Target Audio
Synthesized
Reverb
Audio
Z
Decoder
Filtered Noise AlyAA M/
Loudness
Multi-Scale Spectrogram Loss
Autoencoder architecture. Red components are part of the neural network architecture, green components the latent representation, and yellow components are deterministic synthesiz- ers and effects. Components with dashed borders are not used in all of our experiments. Namely, is not used in the model trained on solo violin, and reverb is not used in the models trained on NSynth. See the appendix for more detailed diagrams of the neural network components.
Figure 2:
4.1 DDSP AUTOENCODER
DDSP components do not put constraints on the choice of generative model (GAN, VAE, Flow, etc.), but we focus here on deterministic autoencoder to investigate the strength of DDSP components independent of any particular approach to adversarial training, variational inference, Jacobian design. Just autoencoders utilizing convolutional layers outperform fully-connected autoencoders on images, we find DDSP components are able to dramatically improve autoencoder performance in the audio domain. Introducing stochastic latents (such as in GAN, VAE, and Flow models) will likely further improve performance, but we leave that to future work as it is orthogonal to the core question DDSP component performance that we investigate in this paper.
In a standard autoencoder, an encoder network fenc() maps the input æ to a latent representation fene and a decoder network faec(-) attempts directly reconstruct the input = fuee Our architecture (Figure 2) contrasts with this approach through the use of DDSP components and a decomposed latent representation.
Encoders: Detailed descriptions of the encoders are given in Section B. For the supervised au- toencoder, the loudness i(t) is extracted directly from the audio, pretrained CREPE model with fixed weights (Kim et al. 2018) is used as an ft) encoder to extact the fundamental frequencyy and optional encoder extracts a time-varying latent encoding =(t) of the residual information. For the (t) encoder, MFCC coefficients (30 per frame) are first extracted from the audio, which corre- spond the smoothed spectral envelope of harmonics (Beauchamp 2007). and transformed by single GRU layer into 16 latent variables per frame.
For the unsupervised autoencoder, the pretrained CREPE model is replaced with a Resnet architec- ture (He et al. 2016) that extracts f(() from mel-scaled log spectrogram of the audio, and is jointly trained with the rest of the network.
The decoder network maps the tuple (f(1), ((t), zt)) control parameters for the additive and filtered noise synthesizers described in Section 3 The synthesizers generate audio based on these parameters, and reconstruction loss between the synthesized and original audio is minimized. The network archi- tecture is chosen to be fairly generic (fully connected, with a single recurrent layer) to demonstrate that it is the DDSP components, and other modeling decisions, that enables the quality of the work.
Decoder: A detailed description of the decoder network is given in Section B.2
Also unique to our approach, the latent f(() is fed directly the additive synthesizer it has structural meaning for the synthesizer outside the context of any given dataset. As shown later in Section 5.2 this disentangled representation enables the model both interpolate within and

---

Published as a conference paper at ICLR 2020
Loudness (L1) (L)
FÜ Outliers
Supervised
WaveRNN (Hantrakul et 2019)
0.10
1.00
0.07
DDSP Autoencoder
0.07
0.02
0.003
Unsupervised
DDSP Autoencoder
0.09
0.80
0.04
Table Resynthesis accuracies. Comparison of DDSP models to SOTA WaveRNN model provided the same conditioning information. The supervised DDSP Autoencoder and WaveRNN models use the fundamental frequency from a pretrained CREPE model, while the unsupervised DDSP autoencoder learns to infer the frequency from the audio during training.
trapolate outside the data distribution. Indeed, recent work support incorporation of strong inductive biases as prerequisite for learning disentangled representations (Locatello et al. 2018).
Model Size: Table B.6 compares parameter counts for the DDSP models and comparable mod- els including GANSynth (Engel et al. 2019), WaveRNN (Hantrakul et al., 2019), and a WaveNet Autoencoder (Engel et al. (017). The DDSP models have the fewest parameters (up 10 times less), despite no effort to minimize the model size in these experiments. Initial experiments with very small models (240k parameters, 300x smaller than WaveNet Autoencoder) have less realistic outputs than the full models, but still have fairly high quality and are promising for low-latency applications, on CPU or embedded devices. Audio samples are available in the online supple- men
4.2 DATASETS
NSynth: We focus on a smaller subset of the NSynth dataset (Engel et al, 2017) consistent with other work (Engel et al. 2019 Hantrakul et al. 2019). It totals 70,379 examples comprised mostly of strings, brass, woodwinds and mallets with pitch labels within MIDI pitch range 24-84. We employ 80/20 train/test split shuffling we use the autoencoder as described above (with the x(1) encoder). We experiment with both the supervised and unsupervised variants.
instrument families. For the NSynth experiments,
Solo Violin: The NSynth dataset does not capture aspects of a real musical performance. Using the MusOpen royalty free music library, collected 13 minutes of expressive, solo violin perfor- mances We purposefully selected pieces from single performer (John Gamerr) that were mono- phonic and shared consistent room environment encourage the model focus performance. Like NSynth, audio is converted to mono 16kHz and divided into second training examples (64000 samples total). Code to process the audio files into a dataset is available online f
For the solo violin experiments, we use the supervised variant of the autoencoder (without the =(t) encoder), and add reverb module the signal processor chain to account for room reverberation. While the room impulse response could be produced as an output of the decoder, given that the solo violin dataset has single acoustic environment, corresponding to 64000 dimensions) for the impulse response.
single fixed variable (4 second reverb
4.2.1 MULTI-SCALE SPECTRAL LOSS
The primary objective of the autoencoder is to minimize reconstruction loss. However, for audio waveforms, point-wise loss the waveform is not ideal, as two perceptually identical audio samples may have distinct waveforms, and point-wise similar waveforms may sound very different.
Instead, we multi- scale spectral loss-similar to the multi-resolution spectral amplitude distance in Wang et al. (magnitude) spectrogram S, and Š,, respectively, with a given FFT size i, and define the loss as the
as follows. Given the original and synthesized audio, we compute their
FFie pieces by John Garner (II. Double, III. Corrente, Double Presto, VI. Double, VIII. Double) from https: ht no- bwv- 1awv-100 1002/

---

Published as a conference paper at ICLR 2020
Loudness
Fundamental Frequency
Spectral Centroid
Figure 3: Separate interpolations over loudness, pitch, and timbre. The conditioning features (solid lines) are extracted from two notes and linearly mixed (dark to light coloring). The features of the resynthsized audio (dashed lines) closely follow the conditioning. On the right, the latent vectors, z(t), are interpolated, and the spectral centroid of resulting audio (thin solid lines) smoothly varies between the original samples (dark solid lines).
sum of the Ll difference between S; and S; as well as the L1 difference between log S; and log S,.
L, ||S S11l1 + a log S; - log S/1/2-
(4)
where a a weighting term set to 1.0 in our experiments. The total reconstruction loss is then the sum of all the spectral losses, Trecomstruction E, L, In our experiments, we used FFT sizes (2048, 1024, 512, 256, 128, 64), and the neighboring frames the Short-Time Fourier Transform (STFT) overlap by 75%. Therefore, the L:'s cover differences between the original and synthesized audios different spatial-temporal resolutions.
RESULTS
5.1
HIGH-TIDELITY SYNTHESIS
As shown in Figure 5 the DDSP autoencoder learns to very accurately resynthesize the solo violin dataset. Again, we highly encourage readers to listen the samples provided in the online supple- men A full decomposition of the components is provided Figure 5 High-quality neural audio synthesis has previously required very large autoregressive models (Oord et al. 2016 Kalchbrenner et al. 2018) or adversarial loss functions (Engel et al., 2019). While amenable to an adversarial loss, the DDSP autoencoder achieves these results with straightforward I.1 spectrogram loss, small amount of data, and a relatively simple model. This demonstrates that the model is able to efficiently exploit the bias of the DSP components, while not losing the expressive power of neural networks.
For the NSynth dataset, we quantitatively compare the quality of DDSP resynthesis with that of oftt--of-the-ar baseline using WaveRNN (Hantrakul et al. 2019). The models comparable as they are trained on the same data, provided the same conditioning, and both targeted towards realtime synthesis applications. In Table [ compute loudness and fundamental frequency (FO) L metrics described in Section of the appendix. Despite the strong performance of the baseline, the supervised DDSP autoencoder still outperforms it, especially in FO L, This is not unexpected, as the additive synthesizer directly uses the conditioning frequency to synthesize audio.
The unsupervised DDSP autoencoder must learn infer its FO conditioning signal directly from the audio. As described in Section B.4, we improve optimization by also adding perceptual loss in the form of pretrained CREPE network (Kim et al. 2018). While not accurate as the supervised DDSP version, the model does fair job at learning to generate sounds with the correct frequencies without supervision, outperforming the supervised WaveRNN model.
5.2 INDEPENDENT CONTROL OF LOUDNESS AND PITCH
Interpolation: Interpretable structure allows for independent control over generative factors. Each component of the factorized latent variables (f) ((t)),((), x(t)) independently alters samples along matching perceptual axis. For example, Figure 3 shows an interpolation between two sound in the loudness conditioning ((t). With other variables held constant, loudness of the synthesized audio closely matches the interpolated input. Similarly, the model reliably matches intermediate pitches between high pitched (() and low pitched
(().. In Table C.2 of the appendix, we quantitatively demonstrate how across interpolations, conditioning independently controls the corresponding char- acteristics of the audio.
8

---

Published as a conference paper at ICLR 2020
Original (Singing)
Timbre Transer (Violin)
Figure 4: Timbre transfer from singing voice to violin. FO and loudness features are extracted from the voice and resynthesized with a DDSP autoencoder trained on solo violin.
With loudness and pitch explicitly controlled by (f(t), ((t), the model should use the residual =(t) to encode timbre. Although architecture and training do not strictly enforce this encoding, we qual- itatively demonstrate how varying z leads to a smooth change timbre. In Figure (3 we use the smooth shift in spectral centroid, or "center of mass" of spectrum, illustrate this behavior.
f(t) directly controls the additive synthesizer and has structural meaning outside the context of any given dataset. Beyond interpolating between datapoints, the model can extrapolate to new conditions not seen during training. The rightmost plot of Figure 7 demonstrates this by resynthesizing clip of solo violin after shifting f(() down an octave and outside the range of the training data. The audio remains coherent and resembles a related instrument such as cello. f(t) is only modified for the synthesizer, as the decoder is still bounded by the nearby distribution the training data and produces unrealistic harmonic content conditioned far outside that distribution.
Extrapolation: As described in Section
5.3 DEREVERBERATION AND ACOUSTIC TRANSFER
Removing reverb in a "blind" setting, where only reverberated audio is available, is standing problem acoustics (Naylor & Gaubitch) 2010). However, benefit of our modular approach to generative modeling is that it becomes possible to completely separate the source audio from the effect of the room. For the solo violin dataset, the DDSP autoencoder is trained with an additional reverb module as shown in Figure 2 and described in Section 3.6 Figure 7 (left) demonstrates that bypassing the reverb module during resynthesis results completely dereverberated audio, similar to recording in an anechoic chamber. The quality of the approach is limited by the underlying generative model, which quite high for our autoencoder. Similarly, Figure (center) demonstrates that we can also apply the learned reverb model to new audio, in this case singing, and effectively transfer the acoustic environment the solo violin recordings.
5.4 TIMBRE TRANSFER
Figure demonstrates timbre transfer, converting the singing voice of author into violin. PO and loudness features are extracted from the singing voice and the DDSP autoencoder trained on solo violin used for resynthesis. To better match the conditioning features, first shift the fundamental frequency of the singing up by two octaves to violin's typical register. Next, we transfer the room acoustics of the violin recording (as described in Section 5.3) to the voice before extracting loudness, to better match the loudness contours of the violin recordings. The resulting audio captures many subtleties of the singing with the timbre and room acousties of the violin dataset. Note the interesting "breathing" artifacts in the silence corresponding to unvoiced syllables from the singing.
6 CONCLUSION
The DDSP library fuses classical DSP with deep learning, providing the ability to take advantage of strong inductive biases without losing the expressive power of neural networks and end-to-end

---

Published as a conference paper at ICLR 2020
learning. We encourage contributions from domain experts and look forward to expanding the scope of the DDSP library to a wide range of future applications.
REFERENCES
S. O. Arik, H. Jun, and G. Diamos. Fast spectrogram inversion using multi-head convolutional neural networks. IEEE Signal Processing Letters, 26(1):94-98, Jan 2019. doi: 10.1109/LSP. 2018.2880284
James W Beauchamp. Analysis, synthesis, and perception of musical sounds. Springer, 2007.
Merlijn Blaauw and Jordi Bonada. A neural parametric singing synthesizer modeling timbre and expression from natural songs. Applied Sciences, 7(12):1313, 2017.
Pritish Chandna, Merlijn Blaauw, Jordi Bonada, and Emilia Gomez. Wgansing: A multi-voice singing voice synthesizer based on the wasserstein-gai arXiv preprint 1rXiv:1903.11777 2019.
Taishih Chi, Powen Ru, and Shihab A Shamma. Multiresolution spectrotemporal analysis of com- plex sounds. The Journal of the Acoustical Society of America, 118(2)887-906, 2005.
Alexandre Defossez, Neil Zeghidour, Nicolas Usunier, Leon Bottou, and Francis Bach. Sing: cymbol-to-instrumen neural generator. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Basa-Bianchhi,. and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018. URL httt:t/ppp/ppppt nips. cttpaper/ pdf
pp. to
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In International Conference Learning Representations, 2019. URU/attss://ppenrrriiee www.:reoiiii.ee/ooommn mnet/forum? 1d Bd-ByMVTsR5KQ
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in neural information processing systems, pp. 658-666, 2016.
Jesse Engel, Cinjon Resnick, Adam Roberts. Sander Dieleman, Douglas Eck. Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with WaveNet autoencoders. In ICML. 2017.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Guirajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In International Conference on Learn- ing Representations, 2019,
net/ forum www.mmmii-t1xiyn0@m1.
Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, et al. Universal audio synthesizer control with normalizing flows. arXiv preprint 1xXiv:1977000971 2019. 2019..
James L Flanagan. Speech analysis synthesis and perception, volume 3. Springer Science & Busi- ness Media, 2013.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
2018.
Lamtharn (Hanoi) Hantrakul, Jesse Engel, Adam Roberts, and Chenjie Gu. Fast and flexible neural audio synthesis. In ISMIR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matthew D Hoffman and Perry Cook. Feature-based synthesis: Mapping acoustic and perceptual features onto synthesis parameters. In ICMC. Citeseer, 2006.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural nefworks, 2(5):359-366, 1989.
10

---

Published as a conference paper at ICLR 2020
Cheng-Zhi Anna Huang, David Duvenaud, Kenneth C Armold, Brenton Partridge, Josiah W Ober- holtzer, and Krzysztof Z Gajos. Active learning of intuitive control knobs for synthesizers using gaussian processes. In Proceedings of the 19th international conference Intelligent User In- terfaces, pp. 115-124. ACM, 2014.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint arXiv: (tti::8800.0000 0843, 2018.
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165. IEEE, 2018
Anssi Klapuri, Tuomas Virtanen, and Jan-Markus Holm. Robust multipitch estimation for the anal- ysis and manipulation of polyphonic musical signals. In Proc. COST-G6 Conference on Digital Audio Effects, pp. 233 236, 2000.
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and I.. D. Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4):541- 551, 1989, doi: 221162/neco.1999.1.4.2 1989. 1...1.44544 541
URL
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv:1811. 2018.
Robert McAulay and Thomas Quatieri. Speech analysis/synthesis based on a sinusoidal representa- tion. IEEE Transactions Acoustics, Speech, and Signal Processing, 34(4):744-754 1986.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemnn An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837. 2016.
Michelle Moerel, Federico De Martino, and Elia Formisano. Processing of natural sounds in human auditory cortex: tonotopy. spectral tuning, and relation to voice sensitivity. Journal of Neuro- science, 32(41):14205-144 142111 2012.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS Information and Systems, 99(7)1187-1888, 2016.
Patrick A Naylor and Nikolay Gaubitch. Speech dereverberation. Springer Science & Business Media, 2010.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for 2016.
raw audio. arXiv preprint
Heiko Purnhagen and Nikolaus Meine. Hiln-the mpeg-4 parametric audio coding tools. In 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21 Cen- tury. Proceedings (IEEE Cat No. 00CH36353), volume 3, pp. 201-204. IEEE, 2000.
Xavier Serra and Julius Smith. Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition. Computer Music Journal, 14(4):12-2) 12-144 1990.
Julius Orion Smith. Physical audio signal processing: For virtual musical instruments and audio effects. W3K publishing, 2010.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Edwin Tellman, Lippold Haken, and Bryan Holloway. Timbre morphing of sounds with unequal numbers of features. Journal of the Audio Engineering Society, 43(9):678-689, 1995.

---

Published as a conference paper at ICLR 2020
Frédéric E Theunissen and Julie E Elie. Neural processing of natural sounds. Nature Reviews Neuroscience, 15(6):355, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5198-5999099 6008, 2017.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for statisti- cal parametric speech synthesis. arXiv preprint arXiv: 2088, 2019.
Yuxuan Wang. RJ Skerry-Ryan. Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. In INTERSPEECH, 2017.
Werbos. Backpropagation through time: what it does and how do Proceedings of the IEEE, 15500)1550-1166, Oct 1990. doi: 10. 10.1109//.58337.
Ronald J Williams and David Zipser. Gradient-based learning algorithms for recurrent connection- ist networks. College of Computer Science, Northeastern University Boston, MA, 1990.
12

---

Published as a conference paper at ICLR 2020
A
APPENDIX
Full Resynthesis
Combined Audio
Impulse Response
Additive Audio
Filtered Noise Audio
Amplitude
Harmonic Distribution
Noise Magnitudes
Loudness
Fundamental Frequency
Original Audio
Figure 5: Decomposition of clip of solo violin. Audio is visualized with log magnitude spec- trograms. Loudness and fundamental frequency signals are extracted from the original audio. The loudness curve does exhibit clear note segmentations because the effects of the room acous- tics. The DDSP autoencoder takes those conditioning signals and predicts amplitudes, harmonic distributions. and noise magnitudes. Note that the amplitudes are clearly segmented along note boundaries without supervision and that the harmonic and noise distributions are complex and dy- namic despite the simple conditioning signals. Finally, the extracted impulse response applied to the combined audio from the synthesizers to give the full resynthesis audio.
13

---

Published as a conference paper at ICLR 2020
Spectrograms
Synth Parameters
Fundamental Frequency
Amplitude
Harmonic Distribution
Figure 6: Diagram of the Additive Synthesizer component. The synthesizer generates audio as a sum of sinusoids harmonic (integer) multiples of the fundamental frequency. The neural network is then tasked with emitting time-varying synthesizer parameters (fundamental frequency, amplitude, harmonic distribution). In this example finear-frequens magnnitue spectrograms show how the harmonics initially follow the frequency contours the fundamental. We then factorize the harmonic amplitudes into overall amplitude envelope that controls the loudness, and normalized distribution among the different harmonics that determines spectral variations.
Dereverberation
Acoustics Transfer
Transposition Beyond Training
Original
Transformed Audio
Figure 7: Interpretable generative models enables disentanglement and extrapolation. Spectrograms of audio examples include dereverberation of clip of solo violin playing (left), transfer of the extracted room response to new audio (center), and transposition below the range of training data (right).
B
MODEL DETAILS
B.1
ENCODERS
The model has three encoders: f-encoder that outputs fundamental frequency f(t), I-encoder that outputs loudness I(t), and z-encoder that outputs residual vector =tt).
f-encoder: We use a pretrained CREPE pitch detector (Kim et al. 2018) as the f-encoder to extract ground truth fundamental frequencies (FO) from the audio. We used the "large" variant of CREPE, which has SOTA accuracy for monophonic audio samples of musical instruments. For our super- vised autoencoder experiments, fixed the weights of the f-encoder like (Hantrakul al. 2019). and for our unsupervised autoencoder experiemnts we jointly learn the weights of a resnet model fed log mel spectrograms of the audio. Full details of the resnet architecture show in Table 2
14

---

Published as a conference paper at ICLR 2020
I-encoder: We identical computational steps to extract loudness (Hantrakul al. 2019). Namely, an A- A-weighting of the power spectrum, which puts greater emphasis on higher frequencies, followed by log scaling. The vector is then centered according to the mean and standard deviation of the dataset.
z-encoder: As shown in Figure 8 the encoder first calculates MFCC's (Mel Frequency Cepstrum Coefficients) from the audio. MFCC computed from the mel-spectrooaam the audio with FFT size of 1024, 128 bins of frequency range between 20Hz to 8000Hz, overlap of 75%. We use only the first 30 MFCCs that correspond to smoothed spectral envelope. The MFCCs are then passed through a normalization layer (which has learnable shift and scale parameters) and a 5 512-unit GRU. The GRU outputs (over time) fed to a 512-unit linear layer to obtain z()) The z embedding reported in this model has 16 dimensions across 250 time-steps.
Residual Block
Output Size
kTime
kFreg
FFreg
kiitters
layer norm + relu conv
kystters /
layer norm + relu cony
3
SFreq
kysiters' /
layer norm + relu conv
kiitters
add residual Resnet
Output Size
Krime
kFreg
Freg
k Fitters
LogMellppectrorram
(125, 229,
conv2d
(125, 115, 64)
7
2
64
max pool
(125, 58, 64)
3
2
residual block
(125, 58, 128)
3
128
residual block
(125, 57, 128)
3
128
residual block
(125, 29, 256)
256
residual block
(125, 256)
256
residual block
(125, 29, 256)
3
256
residual block
(125, 512)
3
2
512
residual block
(125, 15, 512)
3
512
residual block
(125, 512)
3
512
residual block
(125, 15, 512)
3
512
residual block
(125, 8, 1024)
1024
residual block
(125, 1024)
1024
residual block
(125, 8, 1024)
3
1024
dense upsample time softplus and normalize
(125, 1, 128)
128
(1000, 1, 128)
(1000, 1, 128)
Table 2: Model architecture for the f(t) encoder using a Resnet on log mel spectrograms. Spectro- grams have frame size of 2048 and hop size of 512, and upsampled at the end to have the same time resoultion as other latents (4ms per a frame). All convolutions use "same" padding and temporal stride of 1. Each residual block uses bottleneck structure (He et al. 2016). The final output a normalized probablity distribution over 128 frequency values (logarithmically scaled be- tween 8.2Hz and 13.3kHz (https: www. insp redacoust ics com/ en /MIDIIII note numbers and center frequencies). The finally frequency value is the weighted sum of each frequency by its probability.
B.2 DECODER
The decoder's input is the latent tuple (ft)), =t)) (250 timesteps). Its outputs the parame- ters required by the synthesizers. For example, the case of the harmonic synthesizer and filtered noise synthesizer setup, the decoder outputs a(I) (amplitudes of the harmonics) for the harmonic synthesizer (note that (() is fed directly from the latent), and H (transfer function of the FIR filter) for the filtered noise synthesizer.
15

---

Published as a conference paper at ICLR 2020
Encoder
Target Audio
Calculate MFCC
Normalize
GRU
Dense
Z
Figure Diagram of the z-encoder.
As shown in Figure 9 we use sshared-bottome architecture, which computes shared embedding from the latent tuple, and then have one head for each of the (a(t). H) outputs.
In particular, we apply separate MLPs each the (f(t), ((t), xt)) input. The outputs of the MLPs are concatenated and passed to 512-umit GRU. We concatenate the GRU outputs with the outputs of the ft) and I(t) MLPs (in the channel dimenssion) and pass it through a final MLP and Linear layer to get the decoder outputs.
Decoder
Harmonic
FO
MLP
Audio
Dense
MLP
Concat
GRU
Concat
MLP
Fittered
Noise Alvy M/
Dense
Loudness
MLP
Figure Diagram of the decoder for the harmonic synthesizer and the filtered noise synthesizer.
The MLP architecture, shown in Figure 10 is standard MLP with layer normalization (tf. contrib. layers. layer norm before the RELU nonlincarity. In Figure g all the MLPs have layers and each layer has 512 units.
MLP
Layer
Layer Norm
Layer Norm
Dense
RELU
Dense
RELU
Dense
RELU
Norm
Figure 10: MLP in the decoder.
B.3 TRAINING
Because all DDSP components differentiable, the model is differentiable end-to-end. Therefore, we can apply any SGD optimizer to train the model. We used ADAM optimizer with learning rate 0.001 and exponential learning decay 0.98 every 10,000 steps.
16

---

Published as a conference paper at ICLR 2020
B.4 PERCEPTUAL LOSS
To help guide the DDSP autoencoder that must predict f(t on the NSynth dataset, we also added an additional perceptual loss using pretrained models, such the CREPE (Kim al. 2018 pitch estimator and the encoder of the WaveNet autoencoder (Engel et al. 2017). Compared to the LI loss on the spectrogram, the activations of different layers these models correlate better with the perceptual quality of the audio. After large-scale hyperparameter search, we obtained our best results by using the L.1 distance between the activations of the small CREPE model's fifth max pool layer with a weighting of 5 x 10 relative to the spectral loss.
B.5 SYNTHESIZERS
Harmonic synthesizer / Additive Synthesis: We use 101 harmonics in the harmonic synthesizer (i.e., a(t)'s dimension is 101). Amplitude and harmonic distribution parameters are upsampled with overlapping Hamming window envelopes whose frame size is 128 and hop size 64. Initial phases are all fixed to zero, as neither the spectrogram loss functions or human perception are sensitive absolute offsets in harmonic phase. We also do not include synthesis elements model DC components to signals as they are inaudible and not reflected in the spectrogram losses.
We force the amplitudes, harmonic distributions, and filtered noise magnitudes be non-negative by applying a sigmoid nonlinearity to network outputs. We find a slight improvement in traning stability by modifying the sigmoid to have scaled output, larger slope by exponentiating, and threshold at a minimum value:
= 2.0 (z) 10 + 10-7
(5)
Filtered noise synthesizer: We use 65 network output channels as magnitude inputs to the FIR filter of the filtered noise synthesizer.
B.6 PARAMETER COUNTS
Model
Parameters
WaveNet Autoencoder (Engel et 2017)
75M
WaveRNN (Hantrakul al. 2019)
23M
GANSynth (Engel al. 2019)
15M
DDSP Autoencoder (Unsupervised)
12M
DDSP Autoencoder (Supervised, NSynth)
7M
DDSP Autoencoder (Supervised, Solo Violin)
6M
DDSP Autoencoder Tiny (Supervised, Solo Violin)
0.24M
Table 3: Parameter counts for different models. All models trained on NSynth dataset except for those marked (Solo Violin). Autoregressive models have the most parameters with GANs requiring less. The DDSP models examined in this paper (which have not been optimized at all for size) require 2 to 3 times less parameters than GANSynth. The unsupervised model has more parameters because of the CREPE (small) (t) encoder, and the autoencoder has additional parameters for the =(t) encoder. Initial experiments with extremely small models (single GRU, 256 units), have slightly less realistic outputs, but still relatively high quality (as be heard in the supplemental audio).
C EVALUATION DETAILS
C.I METRICS
The loudness vector extracted from the synthesized audio and L, dis- tance computed against the input's conditioning loudness vector (ground truth). better model will produce lower In distances, indicating input and generated loudness vectors closely match. Note this distance is not back-propagated through the network training objective.
Loudness L, distance:
17

---

Published as a conference paper at ICLR 2020
FO L1 distance: The FO L1 distance is reported in MIDI space for easier interpretation; average FO L1 of 1.0 corresponds to a semitone difference. We use the same confidence threshold 0.85 in (Hantrakul et al. 2019) to select portions where there was detectable pitch content, and compute the metric only in these areas.
FO Outliers: Pitch tracking using CREPE, like any pitch tracker, is not completely reliable. In- stabilities in pitch tracking, such as sudden octave jumps at low volumes, can result errors not due to model performance and need to be accounted for. FO outliers accounts for pitch tracking imper- fections in CREPE (Kim et al. 2018) vs. genuinely bad samples generated by the trained model. CREPE outputs both FO va ue as well as FO confidence. Samples with confidences below threshold of 0.85 (Hantrakul et al. 2019) are labeled as outliers and usually indicate the sample was mostly noise with no pitch harmonic component. As the model outputs better quality audio, the number of outliers decrease, thus lower scores indicate better performance.
C.2 INTERPOLATION METRICS
Loudness L1 and FO Li are shown for different interpolation tasks in table C.2 In reconstrue- tion, the model is supplied with the standard (J(1) a, =() Loudness (I1) and FO (L1) computed against the ground truth inputs. In loudness interpolation, the model is supplied with (f((),, A. I(L) B; (()x), and Loudness (I1) is calculated using I(() as ground truth instead of a)( For FO interpolation, the model supplied with (f(1),, 1(t) (1)a) and FO (Tn) is calculated using f()) as ground truth instead of f(t) For Z. interpolation the model supplied with (1(t) A. I(t) 2 B) The low and constant Loudness (I,)) and FO (T,) metrics across these interpolations indicate the model is able independently vary these variables without affecting other components.
Task Reconstruction Loudness ((t) interp.
Loudness (Lz)
FO (L,)
0.042
0.060
0.061
0.060
FO f(t) interp.
0.048
0.070
7. (t) interp.
0.063
0.065
Table 4: Loudness and FO metrics for different interpolation tasks.
18

---

Published as a conference paper at ICLR 2020
D OTHER NOTES
D.1 SINUSOIDAL. MODELS
While unconstrained sinusoidal oscillator banks are strictly more expressive than harmonic oscil- lators, we restricted ourselves to harmonic synthesizers for the time being to focus the problem domain. However, this not fundamental limitation of the technique and perturbations such as inharmonicity can also be incorporated to handle phenomena such as stiff strings (Smith 2010)).
D.2 REGULARIZATION
is worth mentioning that the modular structure of the synthesizers also makes it possible define additional losses in terms of different synthesizer outputs and parameters. For example, we may impose an SNR loss to penalize outputs with too much noise if know the training data consists of mostly clean data. We have not experimented too much with such engineered losses, but we believe they can make training more efficient, even though such engineering methods deviates from the end-to-end training paradigm,
19


Reviews:
[{"review_id": "rJlmneOVqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model for audio generation/synthesis  where the model is trained to output time-varying parameters of a vocoder/synthesiser rather than directly outputting audio samples/spectrogram frames. The model is trained by mining an L1 loss between the synthesised audio and the real training audio. Overall, I found the paper to be well written, I found the online supplementary material helpful in getting an intuition for the quality of audio samples generated and to understand the various parts of the proposed architecture. I also found the figures to be extremely  informative, especially figure 2. Although, I think the title of the paper could be updated to something more specific like Differentiable Vocoders or something similar, since the description and experiments very specifically deal with audio synthesis with vocoders, even though the components might be general DSP components. I think the paper presents a reasonable alternative to current autoregressive audio generation methods and should be accepted for publication. \n\nMinor Comments\n\n1. The reference provided for RNNs, Sutskever et. al. 2014, should be supplemented with older references from the 80s when RNNs were first trained with backdrop through time. \n2. “The bias of the natural world is to vibrate.” I am not sure what exactly is meant here. Is it really a “bias” if every object in the universe vibrates? I think the term bias has been overloaded in the paper and is confused with structural/inductive priors. Bias as used in the paper, seems to have several means. It would help the description if the authors cleared up this confusing use of the term. \n3. In Section 1.3, the authors claim that one of the strengths of the proposed method is that the models are “relatively small”, however all discussion of the sizes is relegated to the appendix. It would be helpful to the reader if some model complexity comparisons are presented in the results section and a high level summary is presented at the end of Section 1. \n4. In Section 3, some additional high-level description for the Harmonic plus Noise model (Serra & Smith, 1990) should be provided to motivate the discussion and experiments in the rest of the paper. \n5. Section 4.2 should provide some description of the parameters counts for each of the components considered and how this compares with existing auto-regressive generation algorithms. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJemdEkV9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a framework for for audio generation using oscillators with differentiable neural network type learning. They showcase the usefulness and effectiveness of the approach with several examples such as timbre transfer, dereverberation, changing the room impulse response, pitch extrapolation and so on. I can imagine the proposed learnable oscillator based autoencoders in a variety of applications. \n\nI think that this suggested software library can be useful for a wide range of audio researchers, and I commend the authors for this contribution. It is very nice to see an example of research where we make use of our physical understanding of the sound medium rather than blindly throwing a neural network at the problem. \n\nI have one important question though: how susceptible do you think the system is robust with respect to f0 and loudness encoders? Have you experimented with situations where the f0 and the loudness encoders might fail (such as more non-periodic and noisy signals)? \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeL1QN2qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This very nice paper tackles the integration of of domain knowledge in signal processing within neural nets; the approach is illustrated in the domain of music. \n\nThe proof of concept (audio supplementary material) is very convincing. \n\nThe argument is that a \"natural\" latent space for audio is the spectro-temporal domain. Approaches working purely in the waveform, or in the frequency domains must handle the phase issues. Approaches can learn to handle these issues, at the expense of more data. \nA key difficulty is that the L_2 loss does not match the perception. The authors present a perceptual loss that addresses the point (Eq. 4 - clarify the difference w.r.t. Wang et al.), . A natural question thus is whether applying this loss on e.g. Wavenet, Sample RNN or Wave RNN would solve the problem.\n\nIn short, the contribution is in designing a latent space that accounts for independent components of the music signal (pitch; loudness; reverberation and/or noise), using existing components (oscillators, envelopes and filters), and making them amenable to end-to-end optimization (noting that loudness can be extracted deterministically).\n\nI understand that the auto-encoder is enriched with a FIR filter at its core: the input is mapped into the time-frequency domain; convolved with the output of the neural net H_l, and the result is recovered from the time-frequency domain. \nExplain \"frames x_l to match the impulse responses h_l\".\n\nCare (domain knowledge and trials and errors, I suppose) is exercized in the conversion and recovery (shape and size of the window) to remove undesired effects. \n\nOverall, the approach works in two modes: one where the fundamental frequency is extracted, one where it is learned. I miss this comparison in the audio material, could you tell where to look / hear ? \n\n\nQuestions:\n* NN operate at a slower frame rate (sect. 3.2): how much slower ? How sensitive to this parameter ?\n\nDetail\n* were, end p. 5. A word missing ?  \n* are useful --> is useful ? \n* could produced, p.6 ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel differentiable digital signal processing in audio synthesis. The application is novel and interesting. All the reivewers agree to accept it. The authors are encouraged to consider the reviewer's suggestions to revise the paper.
Decision: Accept (Spotlight)