Year: 2021

Abstract:
Musical expression requires control of both what notes are played, and how they are performed. Conventional audio synthesizers provide detailed expressive controls, but at the cost of realism. Black-box neural audio synthesis and concatenative samplers can produce realistic audio, but have few mechanisms for control. In this work, we introduce MIDI-DDSP a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. Starting from interpretable Differentiable Digital Signal Processing (DDSP) synthesis parameters, we infer musical notes and high-level properties of their expressive performance (such as timbre, vibrato, dynamics, and articulation). This creates a 3-level hierarchy (notes, performance, synthesis) that affords individuals the option to intervene at each level, or utilize trained priors (performance given notes, synthesis given performance) for creative assistance. Through quantitative experiments and listening tests, we demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance, and as a complete system, generate realistic audio from a novel note sequence. By utilizing an interpretable hierarchy, with multiple levels of granularity, MIDI-DDSP opens the door to assistive tools to empower individuals across a diverse range of musical experience.

heading: 1 INTRODUCTION
text: Generative models are most useful to creators if they can generate realistic outputs, afford many avenues for control, and easily fit into existing creative workflows (Huang et al., 2020). Deep generative models are expressive function approximators, capable of generating realistic samples in many domains (Ramesh et al., 2021; Brown et al., 2020; van den Oord et al., 2016), but often at the cost of interactivity, restricting users to rigid black-box input-output pairings without interpretable access to the internals of the network. In contrast, structured models chain several stages of interpretable intermediate representations with expressive networks, while still allowing users to interact throughout the hierarchy. For example, these techniques have been especially effective in computer vision and speech, where systems are optimized for both realism and control (Lee et al., 2021b; Chan et al., 2019; Zhang et al., 2019; Wang et al., 2018a; Morrison et al., 2020; Ren et al., 2020).
For music generation, despite recent progress, current tools still fall short of this ideal (Figure 1, right). Deep networks can either generate realistic full-band audio (Dhariwal et al., 2020) or provide detailed controls of attributes such as pitch, dynamics, and timbre (Défossez et al., 2018; Engel et al., 2019; 2020a; Hawthorne et al., 2019; Wang & Yang, 2019) but not both. Many existing workflows use the MIDI specification (Association et al., 1996) to Conventional DSP synthesizers (Chowning, 1973; Roads, 1988) provide extensive control but make it difficult to generate realistic instrument timbre, while concatenative samplers (Schwarz, 2007) play back high-fidelity recordings of isolated musical notes, but require manually stitching together performances with limited control over expression and continuity.
In this paper, we propose MIDI-DDSP, a hierarchical generative model of musical performance to provide both realism and control (Figure 1, left). Similar to conventional synthesizers and samplers that use the MIDI standard (Association et al., 1996), MIDI-DDSP converts note timing, pitch, and expression information into fine-grained parameter control of DDSP synthesizer modules.
We take inspiration from the hierarchical structure underlying the process of creating music. A composer writes a piece as a series of notes. A performer interprets these notes through a myriad of nuanced, sub-second choices about articulation, dynamics, and expression. These expressive gestures are realized as audio through the short-time pitch and timbre changes of the physical vibration of the instrument. MIDI-DDSP is built on a similar 3-level hierarchy (notes, performance, synthesis) with interpretable representations at each level.
While the efficient DDSP synthesis representation (low-level) allows for high-fidelity audio synthesis (Engel et al., 2020a), users can also control the notes to be played (high-level), and the expression with which they are performed (mid-level). A qualitative example of this is shown in Figure 2, where a given performance on violin is manipulated at all three levels (notes, expression, synthesis parameters) to create a new realistic yet personalized performance.
As seen in Figure 1 (left), MIDI-DDSP can be viewed similarly to a multi-level autoencoder. The hierarchy has three separately trainable modules (DDSP Inference, Synthesis Generator, Expression Generator) and three fixed functions/heuristics (DDSP Synthesis, Feature Extraction, Note Detection). These modules enable MIDI-DDSP to conditionally generate at any level of the hierarchy, providing creative assistance by filling in the details of a performance, synthesizing audio for new note sequences, or even fully automating music generation when paired with a separate note generating model.
It is important to note that the system relies on pitch detection and note detection, so is currently limited to training on recordings of single monophonic instruments. This approach has the potential to be extend to polyphonic recordings via multi-instrument transcription (Hawthorne et al., 2021; Engel et al., 2020b; Bittner et al., 2017) and multi-pitch tracking, which is an exciting avenue to explore for future work. Finally, we also show that each stage can be made conditional on instrument identity, training on 13 separate instruments with a single model.
For clarity, we summarize the core contributions of this work:
• We propose MIDI-DDSP, a 3-level hierarchical generative model of music (notes, performance, synthesis), and train a single model capable of realistic audio synthesis for 13 different instruments. (Section 3)
• Expression Attributes: We introduce heuristics to extract mid-level per-note expression attributes from low-level synthesis parameters. (Figure 4)
• User Control: Quantitative studies confirm that manipulating the expression attributes creates a corresponding effect in the synthesizer parameters, and we qualitatively demonstrate the detailed control that is available to users manipulating all three levels of the hierarchy. (Table 2 and Figure 2)
• Assistive Generation: Reconstruction experiments show that MIDI-DDSP can make assistive predictions at each level of the hierarchy, accurately resynthesizing audio, predicting synthesis parameters from note-wise expression attributes, and auto-regressively predicting note-wise expression attributes from a note sequence. (Tables 1a, 1b, 1c)
• Realistic Note Synthesis: An extensive listening study finds that MIDI-DDSP can synthesize audio from new note sequences (not seen during training) with higher realism than both comparable neural approaches and professional concatenative sampler software. (Figure 5)
• Automatic Music Generation: We demonstrate that pairing MIDI-DDSP with a pretrained note generation model enables full-stack automatic music generation. As an example, we use Coconet (Huang et al., 2017) to generate and synthesize novel 4-part Bach chorales for a variety of instruments. (Figure 6)
Audio samples of all results and figures are provided in the online supplement1. We highly recommend readers to access the online supplement to the paper.

heading: 2 RELATED WORK
text: Note Synthesis. Existing neural synthesis models allow either high-level manipulation of note pitch, velocity, and timing (Hawthorne et al., 2019; Kim et al., 2019; Wang & Yang, 2019; Manzelli et al., 2018), or low-level synthesis parameters (Jonason et al., 2020; Castellon et al., 2020; Blaauw & Bonada, 2017). MIDI-DDSP connects these two approaches by enabling both high-level note controls and low-level synthesis manipulation in a single system.
1https://midi-ddsp.github.io/
Most related to this work is MIDI2Params (Castellon et al., 2020), a hierarchical model that autoregressively predicts frame-wise pitch and loudness contours to drive the original DDSP autoencoder (Engel et al., 2020a). MIDI-DDSP builds on this work by adding an additional level of hierarchy for the note expression, training a new more accurate DDSP base model, and explicitly modeling the synthesizer coefficients output by that model, rather than the pitch and loudness inputs to the model. We extensively compare to our reimplementation of MIDI2Params as a baseline throughout the paper.
Hierarchical Audio Modelling. Audio waveforms have dependencies over timescales spanning several orders of magnitude, lending themselves to hierarchical modeling. For example, Dieleman et al. (2018) and Dhariwal et al. (2020) both choose to encode audio as discrete latent codes at different time resolutions, and apply autoregressive models as priors over those codes. MIDI-DDSP applies a similar approach in spirit, but constructs a hierarchy based on semantic musical structure (note, performance, synthesis), allowing interpretable manipulation by users.
Expressive Performance Analysis and Synthesis. Many prior systems pair analysis and synthesis functions to capture expressive performance characteristics (Canazza et al., 2004; Yang et al., 2016; Shih et al., 2017). Such methods often use heuristic functions to generate parameters for driving synthesizers or selecting and modifying sample units. MIDI-DDSP similarly uses feature extraction, but each level is paired with a differentiable neural network function that directly learns the mapping to expression and synthesis controls for more realistic audio synthesis.

heading: 3 MODEL ARCHITECTURE

heading: 3.1 DDSP SYNTHESIS AND INFERENCE
text: Differentiable Digital Signal Processing (DDSP) (Engel et al., 2020a) enables differentiable audio synthesis by using a harmonic plus noise model (Serra & Smith, 1990). Full details are provided in Appendix B.1. Briefly, an oscillator bank synthesizes a harmonic signal from a fundamental frequency f0(t), a base amplitude a(t), and a distribution over harmonic amplitudes h(t), where the dimensionality of h is the number of harmonics. The noise signal is generated by filtering uniform noise with linearly spaced filter banks, where η(t) represents the magnitude of noise output from each filter in time. In this study, we use 60 harmonics and 65 noise filter banks, giving 127 total synthesis parameters each time frame (s(t) = (f0(t), a(t),h(t),η(t))). The final audio is the addition of harmonic and noise signals.
Since the synthesis process is differentiable, Engel et al. (2020a) demonstrate that it is possible to train a neural network to predict the other synthesis parameters given f0(t) and the loudness of the audio, and optimize a multi-scale spectral loss (Wang et al., 2019; Engel et al., 2020a) of the resynthesized audio (Figure 3 left). f0(t) is extracted by a pre-trained CREPE model (Kim et al., 2018), and the loudness is extracted via an A-weighting of the power spectrum (Hantrakul et al., 2019; McCurdy, 1936).
We extend this work for our DDSP Inference module, by providing and additional input features a log-scale Mel-spectrogram of the audio, that produces higher quality resynthesis (Table 1a). Full architectural details are provided in Appendix B.2.

heading: 3.2 EXPRESSION CONTROLS
text: We aim to model aspects of expressive performance with a continuous variable. For example, this enables a performer to choose how loud the note should be performed, or how much vibrato to apply. We define six expression controls (detailed in Appendix B.3), scaled within [0, 1]. These are extracted from synthesis parameters s(t) and applied within the ith note, ni(t), in a note sequence:
Volume: Controls the volume of a note, extracted by taken average amplitude over a note.
Volume fluctuation: Determines the magnitude of a volume change across a note. Used with the volume peak position, below, this can make a note crescendo or decrescendo. This is extracted by calculating the standard deviation of the amplitude over a note.
Volume peak position: Controls where, over the duration of a note, the peak volume occurs. Zero value corresponds to decrescendo notes, whereas one corresponds to crescendo notes. The volume peak position is extracted by calculating the relative position of maximum amplitude in the note.
Vibrato: Controls the extent of the vibrato of a note. Vibrato is a musical technique defined by pulsating the pitch of a note. Vibrato is extracted by applying Discrete Fourier Transform (DFT) on the fundamental frequency f0(t) in a note and take the peak amplitude.
Brightness: Controls the timbre of a note where a higher value corresponds to larger high-frequency harmonic. Brightness is extracted by calculating the average harmonic centroid of a note.
Attack Noise: Controls how much noise occurs at the start of the note (the attack), e.g., the fluctuation of string and bow. At certain settings, this determines whether two notes sound consecutively or separately. The attack noise is extracted by taking a note’s average noise magnitude in the first ten frames (40ms).

heading: 3.3 SYNTHESIS GENERATOR
text: Given the output of the per-note Expression Controls, ei for i = 1, ..., I notes, and a corresponding note sequence, ni, the Synthesis Generator predicts the frame-level synthesis parameters that, in turn, generate audio. Note expression controls are unpooled (repeated) over the duration of the corresponding note to make a conditioning sequence, c(t) = [(e1,n1), ..., (eI ,nI)], with the same length as the fundamental frequency curve, f0(t).
The Synthesis Generator, gθ, is an autoregressive recurrent neural net (RNN) is used to predict a fundamental frequency, f̂0(t) given conditioning sequence, and a convolutional generative adversarial network (GAN), gφ, is used to predict the other synthesis parameters given conditioning sequence and generated fundamental frequency:
f̂0(t) = gθ(c(t)), â(t), ĥ(t), η̂(t) = gφ(c(t), f̂0(t)), (1)
where θ denotes trainable parameters in the autoregressive RNN, and φ indicates trainable parameters in the convolutional GAN. Architectural details for both of these details is provided in Appendix B.4. The autoregressive RNN is trained using cross-entropy loss Lce. The generator of the convolutional GAN is trained by a multi-scale spectral loss Lspec (Eq. 12) and an adversarial objective consisting of a least-squares GAN (LSGAN) Llsgan (Mao et al., 2017) loss and a feature matching loss Lfm (Kumar et al., 2019) (Eq. 15 to Eq. 18). Thus, the total loss applied to the Synthesis Generator can be written as:
L = (Lce + Lspec) + α(Llsgan + γLfm). (2)
In training, the ground-truth f0(t) is input to the convolutional GAN, thus there is no gradient from the convolutional GAN into the autoregressive RNN.

heading: 3.4 EXPRESSION GENERATOR
text: The Expression Generator uses an autoregressive RNN to predict note expression controls from note sequence (Appendix B.6). A single-layer bidirectional GRU extracts context information from input, and a two-layer autoregressive GRU generates note expression. The Expression Generator is trained by mean square error (MSE) loss between ground-truth note expression and teacher-forced prediction (Figure 3 right). At inference time, the output note expression is generated autoregressively and deterministically.
The note sequence used to train the Expression Generator can either be extracted or comes from human labels. To show the full potential of MIDI-DDSP, we use the ground-truth note boundary label from dataset in all experiments for best accuracy. However, note transcription models can be used to provide the note labels.

heading: 4 EXPERIMENTS
text: The structured hierarchy and explicit latent representations used in MIDI-DDSP benefit music control as well as music modeling. We design a set of experiments to answer the following questions: First, does the system generate realistic audio, and if so, how does each module contribute? How does this compare to existing systems? And, second, is the system capable of enabling note-level, performance-level, and synthesis-level control? How effective are these controls? We encourage readers to listen to the samples provided in the online supplement.

heading: 4.1 DATASET
text: To demonstrate modeling a variety of instruments, we use the URMP dataset (Li et al., 2018), a publicly-available audio dataset containing monophonic solo performances of a variety of instruments URMP is widely used in music synthesis research (Bitton et al., 2020; Hayes et al., 2021; Zhao et al., 2019; Engel et al., 2020b). The URMP dataset contains solo performance recordings and their ground truth note boundary of 13 string instruments and wind instruments, which allows
us to test generalization to different instruments. The recordings in the URMP dataset are played by students, and the performance quality is substantially lower compared to virtuoso datasets used in other work (Hawthorne et al., 2019). The URMP dataset contains 3.75 hours of 117 unique solo recordings, where 85 recordings in 3 hours are used as the training set, and 35 recordings in 0.75 hours are used as the hold-out test set.

heading: 4.2 MODEL ACCURACY
text: Modules in MIDI-DDSP can accurately reconstruct output at multiple levels of the hierarchy (Figure 3). We evaluate the reconstruction quality of MIDI-DDSP by evaluating each module. Results are shown in Table 1.
DDSP Inference We measure the difference between reconstruction and ground-truth in the audio spectral loss for our DDSP Inference module and compared it with the original DDSP autoencoder. As shown in Table 1a, with an additional CNN to extract features, the DDSP Inference module can reconstruct audio more accurately than the original DDSP Autoencoder.
Synthesis Generator We predict synthesis parameters from ground-truth note expression and then extract note expression back from the generated synthesis parameters. We measure the root mean square error (RMSE) between note expressions. The prior approach MIDI2Params directly generates synthesis parameters from notes and does not have access to note expressions. However, we can extract note expressions from the generated synthesis parameters and compare them to ground truth. As shown in Table 1b, the Synthesis Generator can faithfully reconstruct the input note expression, whereas without access to note expression, MIDI2Params generates larger error.
Expression Generator We take ground-truth MIDI and evaluate the likelihood of the ground-truth note expressions under the model. As the Expression Generator is auto-regressive, we use teacherforcing to sequentially accumulate the squared error note by note. The total error thus computed can be interpreted as a log-likelihood. We again compare to MIDI2Params, where we auto-regressively condition its own output within and on ground-truth across notes to obtain a note-wise metric. That
is, MIDI2Params sees the ground truth of past notes, but sees its own output for the current note. As shown in Table 1c, the Expression Generator can accurately predict the note expression. In comparison, MIDI2Params without performance-level modeling suffers from predicting the note expression on a higher level when compared to the frame-wise sequence models.

heading: 4.3 AUDIO QUALITY EVALUATION BY HUMANS
text: We evaluate the audio quality of MIDI-DDSP via a listening test. We compare ground truth audio from the URMP dataset to MIDI-DDSP and four other sources: a stripped down version of our system, containing just our DDSP Inference module (Section 3.1), MIDI2Params (Castellon et al., 2020), and two concatenative samplers: FluidSynth and Ableton (detailed in Appendix D.1). DDSP-inference infers synthesis parameters from the ground truth audio; it serves as an upper bound on what is attainable with MIDI-DDSP, which has to predict expression and synthesis parameters from MIDI. MIDI2Params is prior work that synthesizes audio from MIDI by predicting frame-wise loudness and pitch contour, which is fed as input to a DDSP autoencoder.
Participants in the listening test were presented with two 8-second clips, and asked which clip sounded more like a person playing on a real violin, on a 5-point Likert scale. We collected 960 ratings, with each source involved in 320 pair-wise comparisons. Figure 5 shows the number of comparisons in which each source was selected as more realistic. According to a post-hoc analysis using the Wilcoxon signed-rank test with Bonferroni correction (with p < 0.01/15), the orderings shown in Figure 5 (right) are all statistically significant with the exception of ground truth versus DDSP Inference and MIDI2Params versus FluidSynth (Table 6). In particular, MIDI-DDSP was significantly preferred over MIDI2Params, Ableton and FluidSynth.
The difference among the sources can also be seen from visual inspection of the spectrograms (Figure 5, left). While the DDSP Inference module faithfully re-synthesizes the ground-truth audio, MIDI-DDSP generates coherent performance from a series notes and has rich, varying expressions across the notes. MIDI2Params failed to generate coherent expression within a note, generating unrealistic pitch and loudness contours. Also, MIDI2Params stopped the note in the middle when generating the fifth note, suggesting that such a frame-wise generation model is limited in modeling long-term dependency even inside a single note. On the contrary, the note expression modeling in MIDI-DDSP allows it to model dependency at the granularity of the note sequence and use synthesis parameters to model the frame-wise parameter changing inside a single note. The two concatenative synthesizers Ableton and FluidSynth generate the same note expression with identical vibrato and volume for all notes. Although the expression is coherent inside a single note, it fails to generate expression dependency among notes automatically.

heading: 4.4 EFFECTS OF NOTE EXPRESSION CONTROLS
text: To evaluate the behavior of the note expression controls, we measure how well each control correlates with itself after a roundtrip through synthesis. That is, for each sample in the test set, we interpolate the control from lowest (0) to highest (1) in an interval of 0.1 and generate synthesis parameters. Then we extract the note expressions from these synthesis parameters. Table 2 reports the
correlation between the value we put in and the value observed after synthesis. All controls exhibit strong correlation as desired, except for volume peak position. A low correlation may stem from characteristics of the instrument, or imbalances of those performance techniques in the dataset.
Figure 4 illustrates how each note expression affects properties of the sound. For example, as we increase vibrato, we see stronger fluctuations in pitch. Similarly, changing the volume peak position changes the shape of the amplitude curve.

heading: 4.5 FINE GRAINED CONTROL OR FULL END-TO-END GENERATION
text: The structured modelling approach of MIDI-DDSP enables end users to have as much or as little control over the output as they want. A user can add manipulations at certain levels of the hierarchy or let the model guide the synthesis.
On one end of this spectrum, Figure 2 shows the results of an end user manipulating each level of MIDI-DDSP. Because different levels of the MIDI-DDSP hierarchy correspond with different musical attributes, a user can make manipulations at the note-level to change the attack noise and volume to create staccato notes (second green box in Figure 2) or a user could make adjustments to the synthesis-level to control the pitch contour for making a “pitch bend” (yellow box in Figure 2).
On the other end of the spectrum, MIDI-DDSP can be paired with generative symbolic music models to make fully generated, realistic end-to-end performances. As shown in Figure 6, MIDI-DDSP can be combined with a composition Bach chorales model COCONET (Huang et al., 2017), to form a fully generated musical quartet that sounds like real instruments performance. Readers are encouraged to listen to both the hand-tuned and end-to-end performances on our accompanying website.

heading: 5 CONCLUSION
text: We proposed MIDI-DDSP, a hierarchical music modeling system that factorizes the generation of audio to note, performance, and synthesis levels. By proposing explicit representations for each level alongside modeling note expression, MIDI-DDSP enables effective manipulation and realistic automatic generation of music. We show, experimentally, that the input controls for MIDI-DDSP are correlated with desired performance characteristics (e.g., vibrato, volume, etc). We also show that listeners preferred MIDI-DDSP over existing systems, while enabling fine-grained control of these characteristics. MIDI-DDSP can also connect to other models to construct a full audio generation model, where beginners can obtain realistic novel music from scratch, while expert users can manipulate results based on model prediction to realize unique musical design.

heading: A APPENDIX

heading: B MODEL DETAILS
text: B.1 DDSP SYNTHESIZER
The harmonic signal is synthesized using a bank of Kh sinusoidal oscillators parameterized by fundamental frequency f0(t), harmonic amplitudes a(t), and harmonic distribution h(t). The noise signal is synthesized by a filtered noise synthesizer parameterized by noise magnitudes η(t). Due to the strong inductive bias introduced by DDSP, the synthesis parameters are defined with interpretable meanings. For the DDSP synthesis used in this work, s(t) = (f0(t), a(t),h(t),η(t)), where f0(t) ∈ R1×t, a(t) ∈ R1×t, h(t) ∈ R60×t, η(t) ∈ R65×t. Same as the original DDSP synthesizer, the noise signal is generated by filtering uniform noise given noise magnitudes η(t) in Kn frequency bins. An exponential sigmoid nonlinearity is applied to the raw neural network output to generate the a(t), h(t) and η(t): exp-sigmoid(x) = 2.0 · sigmoid(x)log 10 + 10−7. The harmonic distribution h(t) is further normalized to constrain amplitudes of each harmonic component: ∑K k=0 h k(t) = 1.
In this paper, we use 60 harmonic bins to synthesize harmonic signal, and 65 magnitude bins to synthesize filtered noise. The frame size is set to 64 samples per frame, and the sample rate of the audio synthesis is set to 16000. After x(t) is synthesized, a reverb module is applied to x(t) to capture essential reverberation in the environment and instrument. The reverb module is implemented as a frequency domain convolution with a learnable impulse response parameter. This paper uses different reverb parameters for different instruments, and the reverb parameters are learned with back-propagation. The learnable impulse response of the reverb is set to have a length of 48000. In experiments, we found the latter part of the impulse response, which has minimal impact on the timbre and environmental reverb, would cause a very long lingering sound. Thus, in inference time, we add an exponential decay to the impulse response after 16000 samples to constraint the lingering effect:
IR′(t) = IR(t), 0 ≤ t ≤ 16000 IR′(t) = IR(t) · exp (−4(t− 16000)), 16000 < t ≤ 48000
} (3)
where IR′(t) is the original impulse response, and IR′(t) is the impulse response after decay.
B.2 DDSP INFERENCE
In DDSP Inference, a fully-connected network is used on fundamental frequency and loudness, and an 8-layer convolutional neural network (CNN) (Lecun et al., 1998) is used on log-scale Mel-spectrogram to extract features. Then a bi-directional long-short term memory network (LSTM) (Hochreiter & Schmidhuber, 1997) and a fully connected layer are used to map the features to synthesis parameters.
Similar to the original DDSP (Engel et al., 2020a), the fundamental frequency is estimated by CREPE (Kim et al., 2018) model, and the loudness is extracted via an A-weighting of the power spectrum (Hantrakul et al., 2019). In extracting features from the audio, our approach differs from the autoencoder in the original DDSP (Engel et al., 2020a) work. We use a CNN to extract ad-
ditional features from the log-scale Mel-spectrogram to allow better feature extraction with more information for predicting synthesis parameters. Features are extracted by an 8-layer CNN with batch normalization (Ioffe & Szegedy, 2015) and are concatenated with the features extracted by a fully-connected layer from fundamental frequency and loudness. The detailed architecture of the CNN in the DDSP Inference Module is shown in Table 3. The CNN network in the DDSP Inference module is similar to the one used in (Kong et al., 2020) for computational efficiency.
The DDSP Inference Module is optimized via Adam optimizer in a batch size of 16 and a learning rate of 0.0003. We choose a log-scale Mel-spectrogram of 64 dimensions in this work to extract features by CNN. The features extracted by CNN are mapped to a dimension of 256, as well as the features extracted from the fundamental frequency and loudness. If trained in the multi-instrument setting, an instrument embedding in 64 dimensions will be concatenated. The bi-directional LSTM has a hidden dimension of 256.
B.3 EXPRESSION CONTROLS
Given frame-wise extracted synthesis parameters s(t) and a note starts at frame Ton and ends at frame Toff , we define τ ∈ [Ton, Toff ] and total frame duration of the note Tn = Toff − Ton. The synthesis parameters for a single note can be known, namely fundamental frequency f0(τ), amplitude a(τ), harmonic distribution h(τ), and noise magnitude η(τ).
In order to make the scalar note expression controls value better match the perceptual of human, the a(τ), and η(τ) are transformed into log scale:
a′(τ) = 20 log10 a(τ), η ′(τ) = 20 log10 η(τ) (4)
The note expression controls are extracted as follows:
Volume 1
Tn Tn∑ i=1 a′(i). (5)
Volume fluctuation √√√√ 1 Tn Tn∑ i=1 (a′(i)− 1 Tn Tn∑ j=1 a′(j))2. (6)
Volume peak position 1
Tn argmax i a′(i) ∀i ∈ [1, Tn]. (7)
Vibrato Inspired by previous works on expressive performance analysis (Marchini et al., 2014; Li et al., 2015; Yang et al., 2016), the vibrato is calculated by applying Discrete Fourier Transform (DFT) to the fundamental frequency sequence:
max i F{f0(t)}i, (8)
where F{·} defines the DFT function. Only notes with a vibrato rate between 3 to 9 Hz and longer than 200ms are recorded. Otherwise, the vibrato of the note is set to zero. In calculating the DFT function, the fundamental frequency is zero-padded to 1000 frames.
Brightness
b = 1
Tn Tn∑ i 60∑ k=1 k · hk(i), (9)
where hk(i) represents the k-th bin of the harmonic distribution in the i-th time-step.
Attack Noise 1
10 10∑ i=1 65∑ k=1 η′ k (i), (10)
where η′k(i) represents the k-th bin of the noise magnitudes in the i-th time-step.
One can also come up with other note expression controls that describe expression controls in a note. Alternatively, the expression controls can be learned in an unsupervised way by neural networks. However, we leave that for future work.
B.4 SYNTHESIS GENERATOR
The architecture of the Synthesis Generator is shown in Figure 10. In constructing conditioning sequence from note expression and note sequence, frame-wise note expression and note pitch is expanded to frame-wise sequence by repeating the number of frames the note occupies. Then, fame-wise note expression and pitch are concatenated with binary note onsets, offsets, and a scalar note positioning code to provide additional information for note boundary.
The autoregressive RNN for f0(t) generation consists of a single-layer Bi-LSTM and a 2-layer GRU that autoregressively sample the deviation in pitch from note. Wang et al. (2018b) previously
proposed using autoregressive RNN to sample f0(t) for speech. As the f0(t) is generated by adding the deviation from the note pitch, a stacked dilated 1-D convolution (van den Oord et al., 2016) is used to generate the rest of synthesis parameters given both c(t) and f0(t) as condition. The stacked convolution consists of 4 stacks, and each stack has five layers of 1-D convolution with exponentially increasing dilation rate followed by ReLU activation (Maas et al., 2013) and layer normalization (Ba et al., 2016). The autoregressive RNN outputs in the semitone range of [−1, 1]. The final f0(t) is predicted by adding the note pitch in the semitone scale.
The conditioning sequence input is mapped to a dimension of 256 by a linear layer before being sent into an autoregressive RNN and dilated convolution network. If trained on the multi-instrument dataset, a 64-dimension instrument embedding is concatenated after the linear layer. The autoregressive RNN in the Synthesis Generator has a hidden dimension of 256. A dropout of rate 0.5 is applied to all GRU units and autoregressive input during training to avoid overfitting and exposure bias.
The full details of the dilated convolutional network architecture are shown in Table 4.
The architecture of the discriminator used in the Synthesis Generator is shown in Figure 11, and the detailed architecture of the discriminator block is shown in Table 5. The discriminator is motivated by the multi-scale discriminator network in previous works generating waveforms and spectrograms (Kumar et al., 2019; Lee et al., 2021a). It consists of 3 identical conditional discriminator networks running at different scales to learn features of synthesis parameters at different time resolutions. One single discriminator network consists of 4 blocks at each scale, and each block extracts features from predicted synthesis parameters and the conditioning sequence. For each feature stream in a block, two 1-D convolutional layers are used with Leaky-ReLU activation function (Xu et al., 2015), with skip connections and layer normalization (Ba et al., 2016).
The Synthesis Generator is trained by minimizing both reconstruction loss and adversarial loss:
L = Lrecon + αLadv. (11)
The reconstruction loss consists of a cross-entropy loss to train the RNN that predicts fundamental frequency and a multi-scale spectral loss to train the dilated convolution that predicts the amplitude, harmonic distribution, and noise magnitude:
Lrecon = − ∑ i f0i log f̂0i + Lspec. (12)
The multi-scale spectral loss used in DDSP (Engel et al., 2020a) is used for the reconstruction loss. The multi-scale spectral loss computes the L1 difference between Short-Time Fourier Transform (STFT) of the predicted and target audio in different FFT sizes. Given the spectrogram of the predicted audio Ŝi and that of the target audio Si with FFT size i, the multi-scale spectral loss computes the L1 difference between Ŝi and Si as well as log Ŝi and logSi:
L(i)spec = ||Si − Ŝi||1 + β|| logSi − log Ŝi||1, (13)
Lspec = ∑ i L(i)spec ∀i ∈ {2048, 1024, 512, 256, 128, 64}. (14)
The adversarial loss used to train the dilated convolutional network in the Synthesis Generator combines a least-squares GAN (LSGAN) (Mao et al., 2017) objective and a feature matching objective objective (Kumar et al., 2019):
Ladv = Llsgan + γLfm. (15)
Given discriminator networkDk in k-th scale, the LSGAN objective to train the Synthesis Generator can be written as:
Llsgan = Ec  ∑ k=1,2,3 ||Dk(ŝ, c)− 1||2  , (16) and the LSGAN objective for training the discriminator is:
min Dk
E [||Dk(s, c)− 1||2 + ||Dk(ŝ, c)||2] ,∀k = 1, 2, 3. (17)
Given the output of i-th feature map from the k-th discriminator Dk, the feature map matching objective is calculated as L1 difference between corresponding feature map:
Lfm = Es,c [ 4∑ i=1 1 Ni ||D(i)k (s, c)−D (i) k (ŝ, c)||1 ] , (18)
where Ni is the number of units in i-th layer of the feature map.
In training, the gradient from the adversarial loss is blocked before the RNN. That is, the RNN is trained by the cross-entropy loss only. The Synthesis Generator is trained using 4 seconds of audio with a frame length of 4ms, resulting in a sequence length of 1000. The Synthesis Generator is optimized via Adam optimizer in a batch size of 16 and a learning rate of 0.0003, with an exponential learning rate decay at a rate of 0.99 per 1000 steps. The discriminator is optimized using Adam optimizer in a batch size of 16 and a learning rate of 0.0001. α = 1, β = 1, and γ = 10 are used for loss coefficients.
At inference time, the autoregressive RNN in the Synthesis Generator sample fundamental frequency deviation using nucleus sampling (Holtzman et al., 2020) with a p = 0.95 to avoid sudden unrealistic change in fundamental frequency contour.
B.5 IMPROVEMENT OF USING GAN IN SYNTHESIS GENERATOR
Without GAN, the model will suffer from the “over-smoothing” problem where the model generates uniform and smoothed harmonic distribution and noise magnitudes over the whole note. We show in Figure 12 for a qualitative effect of using the GAN. With the introduction of adversarial training, the proposed model overcomes the over-smoothing problem potentially caused by one-to-many mapping.
B.6 EXPRESSION GENERATOR
In the input of the Expression Generator, the discrete pitch is mapped into a latent space of 64 dimensions via embedding layer, and the scalar duration is mapped into a latent space of 64 dimensions via a fully-connect layer. The input to the bi-directional GRU is a concatenation of two 64 dimension inputs. For the multi-instrument model, an instrument id is additionally mapped to a 64 dimension vector via embedding layer and concatenated to the input. The bi-directional GRU and auto-regressive GRU all use a hidden size of 128 and a dropout rate of 0.5. Input dropout with a rate of 0.5 is applied to the teacher-forced input in training time to avoid over-fitting and exposure bias. The output layer of the Expression Generator uses a two-layer Multi-layer Perceptron (MLP) consists of a fully connected layer with a layer normalization before the ReLU nonlinearity used in (Engel et al., 2020a).
Data augmentation is applied in the training of the expression generator. For the input note sequence, we randomly transpose {−3,−2,−1, 0, 1, 2, 3} semitone(s) and randomly stretch the duration by a factor of {0.9, 0.95, 1, 1.05, 1.1}.
The Expression Generator is trained on a sequence length of 64 notes and a batch size of 256. Adam optimizer (Kingma & Ba, 2014) is used in training with a learning rate of 0.0001. The mean-squareloss (MSE) is used to train the Expression Generator.
B.7 OTHER TRAINING DETAILS
The Expression Generator, Synthesis Generator, and DDSP Inference Module are trained separately. The Expression Generator is trained for 5000 steps, the Synthesis Generator is trained for 40000 steps, and the DDSP Inference Module is trained for 10000 steps.

heading: C DATASET
text: The 13 instruments in the URMP dataset are violin, viola, cello, double bass, flute, oboe, clarinet, saxophone, bassoon, trumpet, horn, trombone, tuba. The recordings in the dataset have a sample rate of 48kHz but are down-sampled to 16kHz to match the sample rate of the DDSP synthesis module. Recordings in the dataset are chunk into segments of 4 seconds with 50% overlap to train the Synthesis Generator.
In the URMP dataset, the solo recordings are part of ensemble pieces. Splitting the same piece played by different instruments into training and test sets can cause data leakage. Thus, we split the dataset based on a random shuffle of the recording but post-hoc adjusted the piece so that the same piece does not appear in both training and test set.

heading: D EXPERIMENT DETAILS
text: D.1 DETAILS OF COMPARING METHODS
We use the orchestral strings pack in Ableton2 without any manual adjustment to synthesize the audio from Ableton. The FluidSynth use the sound font of FluidR3 GM.sf23. The MIDI2Param system proposed by (Castellon et al., 2020) is implemented on our code base following the official implementation on GitHub4. The MIDI2Param system is only trained in the single instrument setting for listening test experiments as practiced by the original paper but is trained in a multi-instrument setting with an add of instrument embedding in other experiments.
D.2 DETAILS OF LISTENING TEST
A Kruskal-Wallis H test of the ratings showed that there is at least one statistically significant difference between the models: χ2(2) = 395.35, p < 0.01. The number of wins for each pair comparison and a Wilcoxon signed-rank test for each pair is shown in Table 6.
2https://www.ableton.com/en/packs/orchestral-strings/ 3https://member.keymusician.com/Member/FluidR3_GM/index.html 4https://github.com/rodrigo-castellon/midi2params

heading: E EXPRESSION ATTRIBUTE CONTROL

heading: F AN EXAMPLE USER EXPERIENCE
text: This section presents a step-by-step user experience of an expert using and adjusting MIDI-DDSP to obtain satisfied music performance. Specifically, the human expert as a violin player will iterate on adjusting note expression controls, starting from what the expression generator predicts for a given piece. Figure 13 shows the original music score ”Viva La Vida” as input to MIDI-DDSP. The synthesized audio is shown in Figure 17.
1. Design and Adjust Articulations The expert first designs the articulations of the piece based on automatic generation, marking up/down bow and the tie between the note, indicating the connections between notes. The results are shown in Figure 14, where ties indicate the notes connected should be played as legato (played consecutively) and should have less attack noise.
The expert adjusts the attack noise based on what the articulations designed. In here, the expert lowers the attack noise of the legato notes. The expert first applies the adjustments to the notes. Then, the expert fine-tuned the attack noise adjustments based on the synthesis result of the earlier adjustment. The synthesized audio is shown in Figure 18.
2. Design and Adjust Fingering The expert designs the fingering based on one’s knowledge and experience of violin performance. The results are shown in Figure 15. The expert designs the first two measures of the piece to be played on E string (”E- - - -...” mark on Figure 15), and assign the fingering of each note (numbers above the notes in Figure 15). The choice of the string and the fingering of notes affects the brightness of the timbre. That is, playing on the E string should have a brighter timbre, and playing on an open string (fingering number ”0”) will have a brighter timbre compared to pressing the string (fingering number other than ”0”).
The expert adjusts the notes’ brightness based on the fingering design, based on the principle described above. The expert will first apply an initial adjustment to the brightness, then fine-tune the number according to the synthesis result. The synthesized audio is shown in Figure 19.
3. Adjust Vibrato The expert adjusts the vibrato of each note. Longer notes will be adjusted to have more vibrato to match how the expert would play the violin. Similarly, the expert first adjusts the note’s vibrato based on the model prediction and fine-tune the numbers based on the synthesis result. The synthesized audio is shown in Figure 20.
4. Design Special Articulations The expert designs special articulations of the notes for artistic purposes. In this example, the expert designed some notes to play as staccato. The results are shown in Figure 16 where the purple dot above the notes indicates it to play as staccato.
To adjust notes to staccato, the volume fluctuation, volume peak position, and attack noise needs to be adjusted from the original value. Specifically, the expert increases the volume fluctuation, decreases the volume peak position, and increases attack noise. Similarly, the expert first adjusts the vibrato of the note and fine-tune the numbers based on the synthesis result. In this step, the expert report that decreasing vibrato can help generate better staccato notes. The synthesized audio is shown in Figure 21.
5. Fine-tune the Piece After all designing and adjustments are made, if needed, the expert takes a final revision of the piece by fine-tuning the expression controls of the notes. In this example, the expert does not fine-tune the piece in the end as the audio synthesized by the previous step is good enough.

heading: G OPEN-SOURCE IMAGE ATTRIBUTION
text: The icons used throughout the paper are used under the Creative Commons license via the Noun Project. We gratefully acknowledge the following creators of these images:
• Audio by cataicon from the Noun Project. • bassoon by Symbolon from the Noun Project. • Clarinet by Symbolon from the Noun Project. • composer by Pham Duy Phuong Hung from the Noun Project. • Flute by Symbolon from the Noun Project. • Neural Network by Ian Rahmadi Kurniawan from the Noun Project. • oboe by Symbolon from the Noun Project. • Synthesizer by Jino from the Noun Project. • Violin by Olena Panasovska from the Noun Project. • Violinist by Luis Prado from the Noun Project.

Review summaries:
This paper proposes a music performance modeling network using three sub-modules which are expression generator, synthesis generator, and DDSP inference. The idea of using these three sub-modules to create three-level performance modeling (the three-level control values are notes, performance features, and synthesis parameters) is interesting and the result shows that the proposed model can give more ability of control while not harming the overall audio generation performance. It seems the proposed method that having three trainable networks with two hand-crafted methods (actually note detection is not the part of the contribution of this paper, so I didn't count it) gives full pipeline of hierarchical modeling and this is a good research contribution.

This paper presents a controllable rendering engine for MIDI files, based on the DDSP framework. Given F0 and loudness contour, DDSP can estimate the parameters of a harmonic + noise synthesis model, to render a corresponding audio file. Similar to MIDI2Params, which predicts framewise FO and loudness contours from a MIDI file, MIDI DDSP introduces an intermediate hierarchical level, allowing to control some newly introduced "expression controls". A mapping from MIDI files to "expression controls" is learnt, so that MIDI files can be automatically rendered. However, because these note wise controls are explicit, they also allow human manipulation of the performance rendering. Influence of the expression controls is assessed with a correlation study, showing that the human manipulation has the expected effect on the generated performance. This quantitative evaluation is further confirmed by convincing audio examples.