MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling
Authors: Yusong Wu
Ethan Manilow
Yi Deng
Rigel Swavely
Kyle Kastner
Tim Cooijmans
Aaron Courville
Cheng-Zhi Anna Huang
Jesse Engel
Venue: ICLR 2022 Oral (2022)
Keywords: Audio Synthesis
Generative Model
Hierarchical
DDSP
Music
Audio
Structured Models

Abstract: Musical expression requires control of both what notes that are played, and how they are performed. Conventional audio synthesizers provide detailed expressive controls, but at the cost of realism. Black-box neural audio synthesis and concatenative samplers can produce realistic audio, but have few mechanisms for control. In this work, we introduce MIDI-DDSP a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. Starting from interpretable Differentiable Digital Signal Processing (DDSP) synthesis parameters, we infer musical notes and high-level properties of their expressive performance (such as timbre, vibrato, dynamics, and articulation). This creates a 3-level hierarchy (notes, performance, synthesis) that affords individuals the option to intervene at each level, or utilize trained priors (performance given notes, synthesis given performance) for creative assistance. Through quantitative experiments and listening tests, we demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance,  and as a complete system, generate realistic audio from a novel note sequence. By utilizing an interpretable hierarchy, with multiple levels of granularity, MIDI-DDSP opens the door to assistive tools to empower individuals across a diverse range of musical experience.

Published as a conference paper at ICLR 2022
MIDI-DDSP: DETAILED CONTROL OF MUSICAL PERFORMANCE VIA HIERARCHICAL MODELING
Yusong Wu', Ethan Manilow Yi Deng", , Rigel Swavely', Kyle Kastner', Tim Cooijmans', Aaron Courville', Cheng-Zhi Anna Huang" 1,4 Jesse Engel*
* Equal Contribution
Mila, Quebec Artificial Intelligence Institute, Université de Montréal, Northweeteern University, "New York University, *Google Brain
wu. yww.nnunnnmma......... quebec
(emanilow, annahuang, jesseengel]@@ooggee.c
ABSTRACT
Musical expression requires control of both what notes are played, and how they are performed. Conventional audio synthesizers provide detailed expressive con- trols, but the cost of realism. Black-box neural audio synthesis and concatena- tive samplers can produce realistic audio, but have few mechanisms for control. In this work, we introduce MIDI-DDSP a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. Start- ing from interpretable Differentiable Digital Signal Processing (DDSP) synthesis parameters, we infer musical notes and high-level properties of their expressive performance (such as timbre, vibrato, dynamics, and articulation). This creates a 3-1evel hierarchy (notes, performance, synthesis) that affords individuals the op- tion to intervene at each level, or utilize trained priors (performance given notes, synthesis given performance) for creative assistance. Through quantitative periments and listening tests, we demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for note sequence, independently manipulate the attributes of a given performance, and as complete system, generate realistic audio from a novel note sequence. By utilizing an in- terpretable hierarchy, with multiple levels of granularity, MIDI-DDSP opens the door to assistive tools to empower individuals across a diverse range musical experience.
INTRODUCTION
Generative models are most useful to creators they can generate realistic outputs, afford many avenues for control, and easily fit into existing creative workflows (Huang et al. (020). Deep gener- ative models are expressive function approximators, capable of generating realistic samples in many domains (Ramesh al. 2021: Brown al. 2020; van den Oord al. 2016), but often at the cost of interactivity, restricting users to rigid black-box input-output pairings without interpretable access the internals of the network. In contrast, structured models chain several stages of interpretable intermediate representations with expressive networks, while still allowing users to interact through- out the hierarchy. For example, these techniques have been especially effective in computer vision and speech, where systems are optimized for both realism and control (Lee et al. 2021b) Chan et al.) 2019 Zhang et al. 2019 Wang et al., 201 Morrison et al. 2020 Ren et al. 2020).
For music generation, despite recent progress, current tools still fall short of this ideal (Figure [
right). Deep networks can either generate realistic full-band audio (Dhariwal et al. 2020 or provide
detailed controls of attributes such as pitch, dynamics, and timbre (Défossez et al. 2018 Engel et al.
'Onine resources:
Code: Dttese///ittuuusccc///ggtttttttt Audio Eaanp//sp.pppp////dddddddd....d
Colab Demo: http:h//hts:::/oaaattta http://co...abr.eeeer.c..oooo
http:///m..pppp///oooo.eeeeeeeee com/eccco////ommmmmmooo/ooocttce http://wwo......ooooooooo///a//a ob/
main/mi

---

Published as a conference paper at ICLR 2022
Notes Pitch. Duration
Note Detection
Expresston Generator
Blackoox Neural Networks
MIDI MIDI- w ddsp
Performance Dynamics, Vierato, Articulation, Timbre
Concalenialvve Sempiers
Realism
Feature Extraction
Synthesis Gonerotor
Synthesis Pitch, Amplitude, Naise, Harmonics (250Hz)
BiBlooa Synthesizers
DDS Inference
DDS Synthesls
Control
Audio
Waveform
(16kHiz))
Figure 1: (Left) The MIDI-DDSP architecture. MIDI-DDSP extracts interpretable features at the performance and synthesis levels, building modeling hierarchy by learning feature generation at each level. Red and blue components indicate encoding and decoding respectively. Shaded boxes represent modules with learned parameters. Both expression features and notes are extracted directly from synthesis parameters. (Right) Synthesizers have wide range of control, but struggle to convey realism. Neural audio synthesis and concatenative samplers can produce realistic audio, but have limited control. MIDI-DDSP enables both realistic neural audio synthesis and detailed user control.
2019 2020a Hawthorne al. 2019 Wang & Yang 2019) but not both. Many existing workflows use the MIDI specification (Association et al. 1996) to Conventional DSP synthesizers (Chowning. 1973 Roads 1988) provide extensive control but make it difficult to generate realistic instrument timbre, while concatenative samplers (Schwarz) 2007) play back high-fidelity recordings of iso- lated musical notes, but require manually stitching together performances with limited control over expression and continuity.
In this paper, we propose MIDI-DDSP, a hierarchical generative model musical performance to provide both realism and control (Figure left). Similar conventional synthesizers and samplers that use the MIDI standard (Association et al. 1996), MIDI-DDSP converts note timing, pitch, and expression information into fine-grained parameter control of DDSP synthesizer modules.
We take inspiration from the hierarchical structure underlying the process of creating music. A composer writes piece series of notes. A performer interprets these notes through myriad of nuanced, sub-second choices about articulation, dynamics, and expression. These expressive ges- tures are realized audio through the short-time pitch and timbre changes the physical vibration of the instrument. MIDI-DDSP is built on a similar 3-level hierarchy (notes, performance, synthesis) with interpretable representations each level.
While the efficient DDSP synthesis representation (low-level) allows for high-fidelity audio synthe- sis (Figell et al., 2020a), users can also control the notes be played (high-level), and the expression with which they performed (mid-level) A qualitative example of this is shown in Figure where a given performance on violin is manipulated at all three levels (notes, expression, synthesis param- eters) create new realistic yet personalized performance.
As seen in Figure (left), MIDI-DDSP can be viewed similarly to a multi-level autoencoder. The hierarchy has three separately trainable modules (DDSP Inference, Synthesis Generator, Expression Generator) and three fixed functions/heuristics (DDSP Synthesis, Feature Extraction, Note Detec- tion). These modules enable MIDI-DDSP conditionally generate any level of the hierarchy,
2

---

Published as a conference paper at ICLR 2022
Automatic
Generation
Adjusted human expert
Pitch bend
More vibrato
Strong attack
Higher octave
Figure 2: An example of detailed user control. Given an initial generation from the full MIDI-DDSP model (top), an expert musician can adjust notes (blue), performance attributes (green), and low- level synthesis parameters (yellow) to craft personalized expression of musical piece (bottom).
providing creative assistance by filling in the details of performance, synthesizing audio for new note sequences, or even fully automating music generation when paired with a separate note gener- ating model.
It is important to note that the system relies on pitch detection and note detection, so is currently limited training on recordings of single monophonic instruments. This approach has the potential to be extend to polyphonic recordings via multi-instrument transcription (Hawthorne et al.) [222]] Engel et al., 2020b, Bittner et al., 2017) and multi-pitch tracking, which is an exciting avenue to explore for future work. Finally, we also show that each stage can be made conditional on instrument identity, training on 13 separate instruments with a single model.
For clarity, we summarize the core contributions of this work:
We propose MIDI-DDSP. a 3-level hierarchical generative model of music (notes, per- formance, synthesis), and train a single model capable of realistic audio synthesis for 13 different instruments. (Section 3)
Expression Atrriuttes: We introduce heuristics to extract mid-level per-note expression attributes from low-level synthesis parameters. (Figure
User Control: Quantitative studies confirm that manipulating the expression attributes cre- ates corresponding effect in the synthesizer parameters, and qualitatively demonstrate the detailed control that is available to users manipulating all three levels of the hierarchy. (Table and Figure 2)
Assistive Generation:
Reconstruction experiments show that MIDI-DDSP make assis- tive predictions at each level of the hierarchy, accurately resynthesizing audio, predicting synthesis parameters from note-wise expression attributes, and regressegsss predicting note-wise expression attributes from a note sequence. (Tables [a] Ib [c)
Realistic Note Synthesis: extensive listening study finds that MIDI-DDSP can synthe- size audio from new note sequences (not seen during training) with higher realism than both comparable neural approaches and professional concatenative sampler software. (Figure
Automatic Music Generation: We demonstrate that pairing MIDI-DDSP with pretrained note generation model enables full-stack automatic music generation. As an example, we Coconet (Huang al. 2017 generate and synthesize novel 4-part Bach chorales for a variety of instruments. (Figure
RELATED WORK
Note Synthesis. Existing neural synthesis models allow either high-level manipulation of note pitch, velocity, and timing (Hawthorne et al. 2019 Kim al. 2019 Wang & Yang 2019 Manzelli al.

---

Published as a conference paper at ICLR 2022
DDSP Inference
Synthesis Generator
Expression Generator
C
L
L
Encode
Decode
Trained model
Ground-truth
Prediction
Loss
Figure 3: Separate training procedures for the three modules in MIDI-DDSP. (Left) The DDSP In- ference module predicts synthesis parameters from audio and trained via an audio reconstruction loss on the resynthesized audio. (Middle) The Synthesis Generator module predicts synthesis pa- rameters from notes and their expression attributes (shown as a 6-dimensional color map) and is trained via reconstruction loss and adversarial loss. (Right) The Expression Generator module autoregressively predicts note expression given a note sequence and is trained with teacher forcing. Encoding processes are shown red, and decoding processes are shown in blue and loss calcula- tions are shown in yellow. Thicker arrows indicate the process that is being trained in each level. Ground-truth data are shown solid frames, while model predictions are shown in dashed frames.
2018), or low-level synthesis parameters (Jonason et al. 2020 Castellon et al. 2020 Blaauw & Bonada 2017). MIDI-DDSP connects these two approaches by enabling both high-level level note trols and low-level synthesis manipulation in single system.
Most related this work is MIDI2Params (Castellon al., 2020), hierarchical model that toregressively predicts frame-wise pitch and loudness contours to drive the original DDSP autoen- coder (Engel al. 2020a).. MIDI-DDSP builds on this work by adding additional level hi- erarchy for the note expression, training a new more accurate DDSP base model, and explicitly modeling the synthesizer coefficients output by that model, rather than the pitch and loudness puts to the model. We extensively compare to our reimplementation of MIDI2Params as baseline throughout the paper.
Hierarchical Audio Modelling. Audio waveforms have dependencies over timescales spanning several orders of magnitude, lending themselves to hierarchical modeling. For example, Dieleman et al. (2018) and Dhariwal et al.] (2020) both choose to encode audio discrete latent codes at different time resolutions, and apply autoregressive models as priors over those codes. MIDI-DDSP applies similar approach in spirit, but constructs hierarchy based semantic musical structure (note, performance, synthesis), allowing interpretable manipulation by users.
Expressive Performance Analysis and Synthesis. Many prior systems pair analysis and synthesis functions to capture expressive performance characteristics (Canazza et al.) 2004 Yang et al. 2016 Shih al. 2017). Such methods often heuristic functions to generate parameters for driving synthesizers or selecting and modifying sample units. MIDI-DDSP similarly uses feature extraction, but each level is paired with differentiable neural network function that directly learns the mapping to expression and synthesis controls for more realistic audio synthesis.
MODEL ARCHITECTURE
3.1 DDSP SYNTHESIS AND INFERENCE
Differentiable Digital Signal Processing (DDSP) (Engel et al. 2020a) enables differentiable audio synthesis by using harmonic plus noise model (Srra & Smith 1990). Full details are provided in Appendix B.I Briefly, an oscillator bank synthesizes a harmonic signal from a fundamental frequency fo(t), base amplitude a(t), and a distribution over harmonic amplitudes h(t), where the dimensionality h the number of harmonics. The noise signal is generated by filtering uniform noise with linearly spaced filter banks, where 17( represents the magnitude of noise output
4

---

Published as a conference paper at ICLR 2022
Volume Fluctuation
Volume Peak
Volume
Vibrato
Brightness
Attack Noise
Position
high
Amplitudi
FO
originel
ma
Maw
Harmonic Centroid
Avg. Noise
Figure 4: In MIDI-DDSP, manipulating note-level expression can effectively change the synthesis- level quantities. We show by taking test-set sample (middle row) and adjusting each expression control value to lowest (bottom row) and highest (upper row), how each synthesis quantities (right- most legend) would change. The dashed gray line each plot indicates the note boundary.
from each filter in time. In this study, we use 60 harmonics and 65 noise filter banks, giving 127 total synthesis parameters each time frame (s(t)= (((()),((t))) h(1,, (())). The final audio the addition of harmonic and noise signals.
Since the synthesis process is differentiable, [ngel al. (2020a) demonstrate that is possible to train a neural network to predict the other synthesis parameters given fo(() and the loudness of the audio, and optimize multi-scale spectral loss (Wang et al.) 2019 Engel al., 2020a) of the resynthesized audio (Figure 3 left). ft( is extracted by pre-trained CREPE model (Kim et al.
2018). and the loudness is extracted via an A-weighting of the power spectrum (Hantrakul al.
2019 McCurdy 1936).
We extend this work for our DDSP Inference module, by providing an additional input features extracted by CNN (Lecun al.) 1998) from log-scale Mel-spectrogram of the audio, that produces higher quality resynthesis (Table lak. Full architectural details are provided Appendix B.2
3.2 EXPRESSION CONTROLS
We aim to model aspects of expressive performance with a continuous variable. For example, this enables performer choose how loud the note should be performed, or how much vibrato apply. We define a 6-dimensional vector, for each note, nt, where each dimension corresponds to one of the six expression controls (detailed in Appendix B.3). scaled within [0, 1]. These are extracted from synthesis parameters stt) and applied within the ith note, n., in note sequence:
Volume: Controls the volume of a note, extracted by taking average amplitude over note.
Volume fluctuation: Determines the magnitude of volume change across note. Used with the volume peak position described below, this can make a note crescendo or decrescendo. This extracted by calculating the standard deviation of the amplitude over a note.
Volume peak position: Controls where, over the duration of a note, the peak volume occurs. Zero value corresponds to decrescendo notes, whereas one corresponds to crescendo notes. The volume peak position extracted by calculating the relative position maximum amplitude the note.
Vibrato: Controls the extent of the vibrato of note. Vibrato is a musical technique defined by pulsating the pitch of a note. Vibrato extracted by applying Discrete Fourier Transform (DFT) on the fundamental frequency fo(t) in note and selecting the peak amplitude.
Brightness: Controls the timbre of a note where higher values correspond to louder high- figh-frequency harmonics. Brightness is determined by calculating the average harmonic centroid of note.
Attack Noise: Controls how much noise occurs at the start of the note (the attack), e.g.. the fluc- tuation of string and bow. Attack noise can determine whether two notes sound consecutively or separately. Attack noise is extracted by taking a note's average noise magnitude in the first ten frames (40ms).
5

---

Published as a conference paper at ICLR 2022
3.3 SYNTHESIS GENERATOR
Given the output of the per-note Expression Controls, e; for i = 1,... / notes, and corresponding note sequence, nh; the Synthesis Generator predicts the frame-level synthesis parameters that, in turn, generate audio. Note expression controls are unpooled (repeated) over the duration of the corresponding note to make a conditioning sequence, c(t) [(e,, n1),-., (er, nr)].. with the same length as the fundamental frequency curve, (t).
The Synthesis Generator, go, is an autoregressive recurrent neural net (RNN) is used to predict fun- damental frequency, fo(!) given conditioning sequence, and a convolutional generative adversarial network (GAN), 9a. is used to predict the other synthesis parameters given conditioning sequence and generated fundamental frequency:
fo(t) = go(((t)), a(t), (()), ((t) go (c(c(t)
(1)
where 0 denotes trainable parameters in the autoregressive RNN, and á indicates trainable param- eters in the GAN. Architectural details for both of these details is provided in Appendix [B.4] The
autoregressive RNN is trained using cross-entropy loss Lee. The generator of the GAN is trained by
multi-scale spectral loss Capec (Eq. 13 14 and an adversarial objective consisting a least-squares
GAN (LSGAN) Lisgan (Mao al. 2017) loss and a feature matching loss fim (Appendix [B.4
Eqs. [5 18) written as:
(Kumar et al. 2019). Thus, the total loss applied to the Synthesis Generator can be
(Coe + Lspec) + (Lisgan + fmm)
where a and y are settable hyperparameters that control the overall GAN loss and feature matching loss, respectively. For training, the ground- truth ft( is input to the GAN, thus there is gradient from the GAN to the autoregressive RNN.
3.4 EXPRESSION GENERATOR
The Expression Generator uses an autoregressive RNN to predict note expression controls from note sequence (Appendix B.6). A single-layer bidirectional GRU extracts context information from input, and a two-layer autoregressive GRU generates note expression. The Expression Generator is trained by mean square error (MSE) loss between ground-truth note expression and teacher-forced prediction (Figure 3 right). The note sequence used to train the Expression Generator can either be extracted or comes from human labels. To show the full potential of MIDI-DDSP, we use the ground-truth note boundary label from dataset in all experiments for best accuracy. However, in future work note transcription models can be used to provide the note labels.
4 EXPERIMENTS
The structured hierarchy and explicit latent representations used in MIDI-DDSP benefit music con- trol as well as music modeling. We design set of experiments to answer the following questions: First, does the system generate realistic audio, and if so, how does each module contribute? How does this compare to existing systems? And, second, the system capable of enabling note-level, performance-level, and synthesis-level control? How effective are these controls?
4.1 DATASET
To demonstrate modeling variety of instruments, we use the URMP dataset (Li et al.) 2018), publicly-vvailable audio dataset containing monophonic solo performances of variety instru- ments. URMP is widely used in music synthesis research (Bitton et al. 2020 Hayes et al. 2021 Zhao et al. 2019 Engel et al., 2020b). The URMP dataset contains solo performance recordings and ground truth note boundaries from 13 string and wind instruments. which allows
test MIDI-DDSP on many different instruments. The recordings in the URMP dataset are played by students, and the performance quality is substantially lower compared to virtuoso datasets used in

---

Published as a conference paper at ICLR 2022
Table 1: Each module in MIDI-DDSP produces high-quality reconstruction and accurate predic-
tion. We show reconstruction accuracy of each MIDI-DDSP module against a comparable method.
Model
Spectral Loss
Model
RMSE
Models
RMSE
DDSP Inference
4.28
Synthesis Generator
0.19
Expression Generator
0.14
Engel et al. (2020a)
5.00
MIDI2Params
0.26
MIDI2Params
0.23
(a)
(h)
(c)
Listening study of pairwise comparisons between methods
DDSP Inference
Ground-ton
Ground-truth
DDSP Inference
MIDI-DDSP
MDDP
Ableton
MIDI2Params
RuidSynth
250
Ableton
MIDI2Params
FluidSynth
Number of wins
Figure 5: (left) Comparing the log-scale Mel spectrograms of synthesis results from test-set note quences, MIDI-DDSP synthesizes more realistic audio (more similar to ground-truth and DDSP In- ference) than prior score-to-audio work MIDI2Params (enlarged in Figure 7 The synthesis quality is also reflected in the listening study (right), where the MIDI-DDSP synthesis is perceived as more realistic than the professional concatenative sampler Ableton and the freely available FluidSynth.
other work (Hawthorne al. 2019). The URMP dataset contains 3.75 hours of 117 unique solo recordings, where 85 recordings in 3 hours are used as the training set, and 35 recordings 0.75 hours used as the hold-out test set.
4.2 MODEL ACCURACY
Modules in MIDI-DDSP can accurately reconstruct output at multiple levels of the hierarchy (Fig- 3re We evaluate the reconstruction quality MIDI-DDSP by evaluating each module, and report the average value across all test set samples in Table
DDSP Inference We measure the difference between reconstruction and ground-truth the audio spectral loss for our DDSP Inference module and compared it with the original DDSP autoencoder. As shown in Table Ial with an additional CNN to extract features, the DDSP Inference module can reconstruct audio more accurately than the original DDSP Autoencoder.
Synthesis Generator We predict synthesis parameters from ground-truth note expression and then extract note expression back from the generated synthesis parameters. We measure the root mean square eтror (RMSE) between note expressions. The prior approach MIDI2Params directly gener- ates synthesis parameters from notes and does not have access to note expressions. However, we can extract note expressions from the generated synthesis parameters and compare them to ground truth. As shown in Table [b] the Synthesis Generator can faithfully reconstruct the input note expression, whereas without access note expression, MIDI2Params generates larger error.
Expression Generator We take ground-truth MIDI and evaluate the likelihood of the ground-truth note expressions under the model. As the Expression Generator is autoregressive, we use teacher- forcing to sequentially accumulate the squared error note by note. The total error thus computed can be interpreted as log-likelihood. We again compare to MIDI2Params, where we autoregressively condition its own output within and ground-truth across notes to obtain note-wise metric. That is, MIDI2Params sees the ground truth of past notes, but sees its own output for the current note. As shown in Table Tc the Expression Generator can accurately predict the note expression. In

---

Published as a conference paper at ICLR 2022
Table 2: The note expression outputs are strongly correlated with input adjustment. The Pearson correlation r- r-values are shown in the table (all entries < 0.0001). The bold numbers indicate Pearson r-value larger than 0.7, which we consider indicate strong correlation between the input control and the respective output quantity. For simplicity, only four instruments are shown. More results can found Table
Volume
Vol. Fluc.
Vol. Peak Pos.
Vibrato
Brightness
Attack Noise
All instruments
.97
.78
.57
.70
.92
.93
Violin
.99
.84
.80
.86
.96
.97
Viola
.98
.74
.70
.82
.98
.97
Cello
.97
.64
.54
.74
.98
.94
Double bass
.98
.85
.34
.84
.99
.95
comparison, MIDI2Params without performance-level modeling suffers from predicting the note expression with higher error when compared to our frame-wise sequence models.
4.3 AUDIO QUALITY EVALUATION BY HUMAN LISTENERS
We evaluate the audio quality of MIDI-DDSP via listening test. We compare ground truth audio from the URMP dataset to MIDI-DDSP and four other sources: a stripped down version of our system, containing just our DDSP Inference module (Section 3.1 MIDI2Params (Castellon al. 2020), and two concatenative samplers: FluidSynth and Ableton (detailed in Appendix D.1p. DDSP Inference infers synthesis parameters from the ground truth audio; it serves as an upper bound on what is attainable with MIDI-DDSP, which has to predict expression and synthesis parameters from MIDI. MIDI2Params is prior work that synthesizes audio from MIDI by predicting frame-wise loudness and pitch contour, which is fed input to a DDSP autoencoder.
Participants in the listening test were presented with two 8-second clips, and asked which clip sounded more like person playing ratings, with each source involved in 320 pair-wise comparisons. Figure shows the number of comparisons in which each source selected as more realistic. According to post-hoc analysis using the Wilcoxon signed-rank test with Bonferroni correction (with < 0.01 /15), the orderings shown in Figure (right) all statistically significant with the exception of ground truth versus DDSP Inference and MIDI2Params versus FluidSynth (Table (6. In particular, MIDI-DDSP was significantly preferred over MIDI2Params, Ableton, and FluidSynth.
real violin,
5-point Likert scale. We collected 960
The difference among the sources can also be seen from visual inspection of the spectrograms (Fig- ure [5, left). While the DDSP Inference module faithfully re-synthesizes the ground-truth audio, MIDI-DDSP generates a coherent performance from series notes, and has rich, varying expres- sions across the notes. MIDI2Params failed generate coherent expressions within note, gener- ating unrealistic pitch and loudness contours. Also, MIDI2Params stopped the note in the middle when generating the fifth note, suggesting that modelling expressive performance only synthesis level is limited in long-term coherence even inside single note. On the contrary, the note expression modeling MIDI-DDSP allows to model dependencies at the granularity the note sequence and synthesis parameters to model the frame-wise parameter changing inside single note. The two concatenative synthesizers Ableton and FluidSynth generate the same note expression with identical vibrato and volume for all notes. Although the expression is coherent inside single note, they fail to generate expression dependencies between different notes automatically.
4.4 EFFECTS OF NOTE EXPRESSION CONTROLS
To evaluate the behavior of the note expression controls, we measure how well each control cor- relates with itself after a roundtrip through synthesis. That is, for each sample in the test set, we interpolate the control from lowest (0) to highest (1) in interval of 0.1 and generate synthesis pa- rameters. Then we extract the note expressions from these synthesis parameters. Table Treporsss the correlation between the value put in and the value observed after synthesis. All controls exhibit
8

---

Published as a conference paper at ICLR 2022
Expression Generator
Synthesis Generator
DDSP Synthesis
CocoNet
Damasoo
Note
Note Expression
Synthesis Parameters
Audio
Sequence
Figure 6: MIDI-DDSP take input from different sources (human other models) by design- ing explicit latent representations each level. A full hierarchical generative model for music can be constructed by connecting MIDI-DDSP with automatic composition model. Here, show MIDI-DDSP taking note input from a score level Bach composition model and automatically syn- thesizing Bach quartet by generating explicit latent for each level in the hierarchy.
strong correlation as desired, except for volume peak position. low correlation may stem from characteristics of the instrument, imbalances those performance techniques in the dataset.
Figure 4 iillustrates how each note expression affects properties of the sound. For example, as we increase vibrato, see stronger fluctuations in pitch. Similarly, changing the volume peak position changes the shape of the amplitude curve.
4.5 FINE GRAINED CONTROL OR FULL END-TO-ENT END GENERATION
The structured modelling approach of MIDI-DDSP enables end users to have much as little control over the output as they want. A user can add manipulations at certain levels of the hierarchy let the model guide the synthesis.
On one end of this spectrum, Figure 2 shows the results of an end user manipulating each level of MIDI-DDSP. Because different levels the MIDI-DDSP hierarchy correspond with different musical attributes, a user can make manipulations at the note-level to change the attack noise and volume to create staccato notes (second green box in Figure 2) or a user could make adjustments to the synthesis-level to control the pitch contour for making "pitch bend" (yellow box in Figure Zp.
the other end the spectrum, MIDI-DDSP can be paired with generative symbolic music mod- els to make fully generated, realistic end-to-end performances. As shown in Figure 6 MIDI-DDSP can be combined with a composition Bach chorales model COCONET (Huang et al. 2017), to form fully generated musical quartet that sounds like real instruments performance. Readers are encour- aged to listen to both the hand-tuned and end-to-end performances on our accompanying website.
5 CONCLUSION
We proposed MIDI-DDSP, a hierarchical music modeling system that factorizes the generation of audio to note, performance, and synthesis levels. By proposing explicit representations for each level alongside modeling note expression, MIDI-DDSP enables effective manipulation and realistic automatic generation of music. We show, experimentally, that the input controls for MIDI-DDSP correlated with desired performance characteristics (e.g., vibrato, volume, etc). We also show that listeners preferred MIDI-DDSP over existing systems, while enabling fine-grained control of these characteristics. MIDI-DDSP also connect to other models to construct full audio gener- ation model, where beginners can obtain realistic novel music from scratch, while expert users can manipulate results based model prediction to realize unique musical design.

---

Published as a conference paper at ICLR 2022
REFERENCES
MIDI Manufacturers Association et al. The complete midi 1.0 detailed specification. Los Angeles, CA, The MIDI Manufacturers Association, 1996.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv.1607.06450, 2016.
Rachel M Bittner, Brian McFee, Justin Salamon, Peter Li, and Juan Pablo Bello. Deep salience representations for f0 estimation polyphonic music. In ISMIR, pp. 63-70, 2017.
Adrien Bitton, Philippe Esling, and Tatsuya Harada. Vector-quantized timbre representation. arXiv preprint arXiv:2007.06:: 2020.
Merlijn Blaauw and Jordi Bonada. A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs. Applied Sciences, 7(12):1313, dee 2017. ISSN 2076-3417. doi:
10.3390/app7121331
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.
Sergio Canazza, Giovanni De Poli, Carlo Drioli, Antonio Roda, and Alvise Vidolin. Modeling and control of expressiveness in music performance. Proceedings of the IEEE, 92(4))686-701, 2004.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Towards realistic midi instrument synthesizers. In NeurIPS Workshop on Machine Learning for Creativity and Design (2020), 2020.
Caroline Chan, Shiry Ginosar. Tinghui Zhou, and Alexei Efros. Everybody dance now. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5933-5942, 2019.
John M Chowning. The synthesis of complex audio spectra by means of frequency modulation. Journal of the audio engineering society, 21(7):526-55) 534,, 1973.
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. Sing: Symbol-to-insttrument neural generator. arXiv preprint arXiv:1810.09758 2018.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint wxXi:20005.00341, 2020.
Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In Advances in Neural Information Processing Systems, pp. 7989-7999, 2018.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. In ICLR, 2019.
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. DDSP: Differentiable digital signal processing. In International Conference on Learning Representations, 2020a.
Jesse Engel, Rigel Swavely. Lamtharn Hantrakul, Adam Roberts, and Curtis Hawthorne. Self- supervised pitch detection by inverse audio synthesis. In International Conference on Machine Learning. Self-supervised Audio and Speech Workshop, 2020b.
Ian J Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, and Y Bengi.. Generative adversarial networks. arxiv e-prints. arXiv preprint arXiv: 1406.2661,
1406, 2014.
Lamtharn Hantrakul, Jesse H Engel, Adam Roberts, and Chenjie Gu. Fast and flexible neural audio synthesis. In ISMIR, pp. 524-530. 2019.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel. and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Represen- fations, 2019.
10

---

Published as a conference paper at ICLR 2022
Curtis Hawthorne, Ian Simon, Rigel Swavely, Ethan Manilow, and Jesse Engel. Sequence-to- sequence piano transcription with transformers. arXiv preprint arXiv:2107.09142, 2021.
Ben Hayes, Charalampos Saitis, and György Fazekas. Neural waveshaping synthesis. arXiv preprint arXiv:2107.05050, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Douglas Eck. Coun- terpoint by convolution. In Proceedings of ISMIR 2017, 2017.
Cheng-Zhi Anna Huang. Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculeseu, and Car- rie J. Cai. Ai song contest: Human-ai co-creation in songwriting. ArXiv, abs/2010.05388, 2020.
Nicolas Jonason, Bob Sturm, and Carl Thomé. The control-synthest approach for making expres- sive and controllable neural music synthesizers. In 2020 Al Music Creativity Conference, 2020.
Jong Wook Kim. Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165. IEFF, 2018.
Jong Wook Kim, Rachel Bittner, Aparna Kumar, and Juan Pablo Bello. Neural music synthesis flexible timbre control. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 176-180. IEEE, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Qiuqiang Kong. Yin Cao, Turab lqbal. Yuxuan Wang. Wenwu Wang, and Mark D Plumbley. Panns: Lcrge-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems, pp. 14910-14921, 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied document recog- nition. Proceedings of the IEEE, 86(1 86(11):2278-2224, 1998. doi: 10. 109/5.726791.
Sang-Hoon Lee, Hyun-Wook Yoon, Hycong-Rae Noh, Ji-Hoon Kim, and Scong-Whai Lee. Multi- spectrogan: High-diversity and high-fidelity spectrogram generation with adversarial style com- bination for speech synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13198 13206, 2021a
Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong. Revisiting hierarchical approach for persistent long-term video prediction. arXiv preprint arXiv:2 201.00446667 2021b.
Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522-55 535. 2018.
Pei-Ching Li, Li Su, Yi-Hsuan Yang, Alvin WY Su, et al. Analysis expressive musical terms in violin using score-informed and expression-based audio features. In ISMIR, pp. 809-815, 2015.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. ieml, volume 30, pp. 3. Citeseer, 2013.

---

Published as a conference paper at ICLR 2022
Rachel Manzelli, Vijay Thakkar, Ali Siahkamari, and Brian Kulis. Conditioning deep generative raw audio models for structured automatic music. In 19th International Society for Music Information Retrieval Conference, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international confer- ence on computer vision, pp. 2794-2802, 2017.
Marco Marchini, Rafael Ramirez, Panos Papiotis, and Esteban Maestre. The of ensemble: machine learning approach to expressive performance modelling in string quartets. Journal of New Music Research, 43(3):303-3177 2014.
RG McCurdy. 1936.
Tentative standards for sound level meters. Electrical Engineering, 55(3):600-633,
Max Morrison, Zeyu Jin, Justin Salamon, Nicholas Bryan, and Gautham Mysore. Controllable neural prosody synthesis. In Proceedings of the Annual Conference of the International Speech Commumication Association, INTERSPEECH, volume 2020-Octob, pp. 4437-4441, 2020. doi:
10.11477/nnerspedd222000999
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tin Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. arXiv preprint arXiv:2006. 04558, 2020.
Curtis Roads. Introduction to granular synthesis. Computer Music Journal, 12(2): 1--33, 1988.
Diemo Schwarz. Corpus-based concatenative synthesis. IEEE signal processing magazine, 24(2): 92-104, 2007.
Xavier Serra and Julius Smith. Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition. Computer Music Journal, 14(4):12-24, 1990.
Chi-Ching Shih, Pei- Pei-Ching Li, Yi-Ju Lin, Yu-Lin Wang, Alvin WY Su, Li Su, and Yi-Hsuan Yang. Analysis and synthesis of the violin playing style of heifetz and oistrakh. In Proceedings of the 20th International Conference on Digital Audio Effects (DAFx-17), 2017.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. 9th ISCA Speech Synthesis Workshop, pp. 125-125, 2016.
Bryan Wang and Yi-Hsuan Yang. Performancenet: Score-to-ause to-uudio music generation with multi-band convolutional residual network. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 11 1174-1181, 2019.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798-8807, 2018a.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Autoregressive neural f0 model for statistical para- metric speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 26(8):1406-141) 14199 2018b.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for sta- tistical parametric speech synthesis. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:402-415. 2019.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. Proceedings of the International Conference on Machine Learning (ICML) Workshop. 2015.
12

---

Published as a conference paper at ICLR 2022
Chih-Hong Yang, Pei-Ching Li, AW Su, Li Su, Yi-Hsuan Yang, al. Automatic violin synthesis using expressive musical term features. Proceedings of the 19th International Conference on Digital Audio Effects (DAFx-16), pp. 1-7. Brno, Czech Republic, 2016.
Jiangning Zhang, Xianfang Zeng, Yusu Pan, Yong Liu, Yu Ding, and Changjie Fan. Faceswapnet: Landmark guided many-to-many face reenactment. arXiv preprint arXiv:1905, 1805, 2, 2019.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 1735-1744, 2019.
13

---

Published as a conference paper at ICLR 2022
A
APPENDIX
Ground-truth
DDSP Inference
MIDI-DDSP
Ableton
MIDI2Params
FluidSynth
Figure The enlarged log-scale Mel spectrograms of synthesis results Figure
B MODEL DETAILS
B.I DDSP SYNTHESIZER
Differentiable Digital Signal Processing (DDSP) (Engel al. 2020a) enables differentiable audio synthesis by using harmonic plus noise model (Serra & Smith 1990). The harmonic signal is syn- thesized using a bank of K₁ sinusoidal oscillators parameterized by fundamental frequency (t(, harmonic amplitudes a(t), and harmonic distribution h(t). The noise signal is synthesized by filtered noise synthesizer parameterized by noise magnitudes (t).. Due to the strong inductive bias introduced by DDSP, the synthesis parameters highly interpretable, i.e., opposed to some high- dimensional learned latent vector producing audio, with DDSP models the network's output is input the harmonic plus noise model, whose parameters interpretable by definition. For the DDSP synthesis used in this work, s(t) = (fo(t), a(t), (tt), mff)). where fo( E IR 1xt a(t) E R !xt, h(t)) 600xt rtt R65xt.
Same as the original DDSP synthesizer, the noise signal is generated by filtering uniform noise given noise magnitudes (7) in K, frequency bins. An exponential sigmoid nonlinearity is applied to the raw neural network output to generate the a(t), htt) and (t): exp-sigmoid(1) 2.0 sigmoid(x:) log + e, where = 10 10-7 The harmonic distribution h(t) is further normalized to constrain amplitudes of each harmonic component: Σk (1) 1
In this paper, we 60 harmonic bins synthesize harmonic signal, and 65 magnitude bins to synthesize filtered noise. The frame size is set to 64 samples per frame, and the sample rate of the audio synthesis to capture essential reverberation in the environment and instrument. The reverb module is imple- mented as frequency domain convolution with learnable impulse response parameter. This paper uses different reverb parameters for different instruments, and the reverb parameters are learned with back- propaapopg...n.. The learnable impulse response of the reverb set to have length of 48000 sample points. In experiments, we found the latter part of the impulse response, which has minimal impact on the timbre and environmental reverb, would cause very long lingering sound. Thus, in inference time, we add an exponential decay to the impulse response after 16000 samples to constraint the lingering effect:
16000 Hz. After xtt) is synthesized, reverb module applied x(t)
(t) = IR(t),
0 16000
}
(3)
IR' (t) = IR(t) exp (4((t 16000)), 16000 < S 48000
14

---

Published as a conference paper at ICLR 2022
Volume +0.00
-0,50
-0.25
+0.25
+0.50
-0.50
Volume Fluctuation
-0.25
+0.00
+0.25
+0.50
Volume
-0.50
-0.25
+0.00
+0.25
+0.50
-0.50
Volume Fluctuation
-0.25
+0.00 2.00
+0.25
+0.50
The effect of modifying the 'Volume' and 'Volume Fluctuation' note expression for sample. Each row shows the amplitudes of the notes when fixing 'Volume Fluctuation' and changing 'Volume', while each column shows the amplitudes the notes when fixing 'Volume' and changing "Volume Fluctuation'. The number indicates the amount of modification to the note expression control where (+0, +0) is the original sample. Upper figure shows the spectrogram and the bottom figure shows the amplitude of the generations. The cartoon the side indicates how the modified feature would change the synthesis parameters. The gray dash line in the bottom figure indicates the note boundaries.
Figure 8:
generations,
where IR(() is the original impulse response, and IR'(1) is the impulse response after decay.
B.2 DDSP INFERENCE
In work proposed by Engel et al. (020a),, an autoencoder is built to reconstruct audio by predicting synthesis parameters from audio features. We refer to this prior work as the "DDSP autoencoder." Given an audio, fundamental frequency is estimated by CREPE (Kim et al. 2018) model, and the loudness is extracted via A-weighting of the power spectrum (Hantrakul et al. 2019). Then, the
15

---

Published as a conference paper at ICLR 2022
Volume Peak Position
Spectrogran
Spectrogram Amplitude
Vibrato +0.00
0.25
+0.25
15.50
.0.50
4.25
+0.00
+0.25
+0.50
FO
Brightness
4.5
0:25
0.00
+0.25
-0
Spectrogram
0.99
-0.00
18.25
Harmonic Centroid
Attack Noise
0.50
0.25
Spectrogram
Average Noise Magnitude
4.50
425
1.000
+0.25
40.5
wackdahnldd mallmh
Figure 9: The effect of modifying different note expression parameters on an existing sample. The number indicates the amount of modification the note expression control where (+0) is the original sample. Upper figure shows the spectrogram of generations, and the bottom figure shows the quan- tity of the generations affected by the control. The cartoon on the side indicates how the modified feature would change the synthesis parameters. The gray dash line in the bottom figure indicates the note boundaries.
16

---

Published as a conference paper at ICLR 2022
CREPE
FO
Dense
STFT & A-W A-Weighting
Amplitudes
Audio
Loudness
Bi-LSTM
STFT-Mel & 8 Blayerr CNN
Harmonic Distribution
Instrument
Noise Magnitude
Embeddling
Synthesis Parameters
DDSP Inference
Figure 10: The architecture of the DDSP Inference. The DDSP Inference module extract fo and loudness from audio, and use an 8-layer CNN to extract features from Mel-spectrogram. A directional LSTM takes in all features from audio and predict synthesis parameters.
fundamental frequency and loudness are input to a multi-layer perceptron (MLP) respectively. The output of the MLPs concatenated and passed to a uni-directional GRU. Finally, an MLP is used to predict synthesis parameters from GRU output.
Our DDSP Inference module differs from the above DDSP autoencoder by enabling more informa- tion extracted from audio input. The architecture of the DDSP Inference shown in Figure I0] In addition to fundamental frequency estimated by CREPE and loundess extracted via A-weighting of the power spectrum, am 8-layer convolutional neural network (CNN) (Lecun et al.) 1998) is used extract features from log-scale Mel-spectrogram, and a bi-directional long-short network (LSTM) (Hochreiter Schmidhuber, 1997) is then applied to extract contextual features. The use of CNN applied on log-scale Mel-spectrogram help model extract more information from audio input, thus enabling more accurate synthesis parameter estimation.
memory
In DDSP Inference module, fully-connected network is applied on fundamental frequency and loudness. The output is concatenated with the output of CNN, send to the bi-directional LSTM to extract contextual features. Another fully connected layer are used to map the features synthesis parameters. The CNN network in the DDSP Inference module is similar to the one used by Kong al. 2020) for computational efficiency. The detailed architecture of the CNN is shown in Table
The DDSP Inference Module is optimized via Adam optimizer in a batch size of 16 and learning rate 3e - 4. We choose a log-scale Mel-spectrogram with 64 frequency dimensions in this work to extract features by CNN. The features extracted by CNN are mapped to 256 dimensions, and concatenated the features extracted from the fundamental frequency and loudness. When trained in the iuntiiinsttuument setting, 64 dimensional instrument embedding is also concatenated to the aforementioned feature vectors. The bi-directional LSTM has hidden dimension of 256.
B.3 EXPRESSION CONTROLS
Given frame-wise extracted synthesis parameters s(t) for the ith note, n,, in note sequence, say that the ith note starts at frame Ton and ends at frame Tol-- We define 7 E [Ton: Toil to be every frame that note is active and the total frame duration of the note is T, Tor Ton- The synthe- sis parameters for the whole duration of the note are defined by fundamental frequency fo(T) contour, an amplitude contour a(T), a harmonic distribution I(()), and set of noise magnitudes (7)).
The amplitude contour, a(7), and noise magnitudes 17(T) are transformed into dB scale:
(T) 20 log a(T), r7((7) log (T()
(4)
The note expression controls extracted as follows:
17

---

Published as a conference paper at ICLR 2022
ConvBlock
Kernel Size
Stride
Filter Size
Conv2d
(3,3)
(1,1)
Kpitters
Batch norm ReLU
Conv2d
(3.3)
(1,1)
Katters
Batch norm + ReLU
Average pooling
(1,2)
Mel-CNN
Output Size
Filter Size
Dropout Rate
LogMelSpeetrogram
(1000, 64,
ConvBlock
(1000, 32, 64)
64
Dropout
(1000, 32, 64)
0.2
ConvBlock
(1000, 16, 128)
128
Dropout
(1000, 16, 128)
0.2
ConvBlock
(1000, 8, 256)
256
Dropout
(1000, 8, 256)
0.2
ConyBlock
(1000, 4, 512)
512
Reshape
(1000, 2048)
Dense
(1000, 256)
256
Table 3: The architecture of the Mel-CNN (bottom) used to extract features from log-scale Mel- spectrogram in the DDSP Inference module. The Mel-CNN uses convolutinal blocks defined in the first table below.
Volume is the sum of the amplitude contour (in dB) normalized over the length of the note, i.e., the mean amplitude over the note,
Tw
1
2 d'i))
(5)
Tn
Volume fluctuation measures the standard deviation of the amplitude curve (in dB):
T, (
(6)
where d'(r) is the mean amplitude over the whole note.
Volume peak position is the location (in time) of the highest amplitude value normalized over the length of the note,
Tr arg max d' (i) Vi e [1.T,].
(7)
Vibrato is inspired by previous works on expressive performance analysis (Marchini et al. 2014 Liet al. 2015 Yang al. 2016). The vibrato is calculated by applying Discrete Fourier Transtorm
(DFT) to the fundamental trequency sequence:
max F(f((t)).
(8)
where FF-) defines the DFT function. Only notes with a vibrato rate between Hz and longer than 200ms are recorded. Otherwise, the vibrato of the note is set to 0. In calculating the DFT function, the fundamental frequency, fo(t), is zero-padded to 1000 frames.
Brightness is defined as the spectral centroid (in bin numbers) of the harmonic distribution,
T
h
Σ
l
(²),).
(9)
T,
R=
18

---

Published as a conference paper at ICLR 2022
FO
Discriminator
RNN
Avg. Pooling
Conditioning Sequence
Dense
Amplitudes
Discriminator
Avg. Pooling
Instrument
Harmonic
Embedding
Distribution
Discriminator
1D Dilated Conv
Noise
Magnitude
Synthesis Generator
Synthesis Parameters
Discriminator
Figure 11: The architecture of the Synthesis Generator. The Synthesis Generator is GAN whose generator (left) takes in per-note Expression Controls and instrument embedding as conditioning sequence (red box, left) and produces DDSP synthesis parameters, i.e., fo. Amplitudes, Harmonic Distribution, and Noise Magnitudes (green boxes, middle). These synthesis parameters get turned into audio by the DDSP modules.
where h& h*i)) represents the k-th bin the harmonic distribution, h(T) in the i-th time-step, and we // to refer the number of bins in the harmonic distribution used by the DDSP module (we use h = 60, see Appendix B.2).
Attack Noise refers to the amount of noise that occurs at the beginning of note. Many instruments have a high amount of noise in the first few milliseconds of a note (e.g., a bow scraping across violin string), before the harmonic components are heard. We determine attack noise like
(1),
(10)
where N determines how many of the first few frames we look at to determine the attack noise (we set N 10, corresponding to 40ms), (i) represents the k-th bin of the dB-scaled noise magnitudes in the i-th time-step, and |m) is the number of noise magnitude bins in the DDSP module (set to |n| 65, see Appendix [B.2).
Recall that all expression controls are normalized to be in the interval [0.0, 1.0). concatenated to 6-dimensional vector, and repeated for the full frame duration of the note, Tú, before we use them with the MIDI-DDSP networks. See Sections 3.2 and 3.3 for additional information. We note that these not the only ways determine the "expression" of note, definitive. Expression does not even need to be hand designed, and could perhaps be learned in an unsupervised way by neural networks. However, we designed our expression controls because they are all inherently interpretable (compared to black box neural net features). We leave the exploration of other ways to define expression for future work.
definitions
For constructing conditioning sequence from these note expressions and an input note sequence, the frame-war wise note expressions and note pitch is expanded these parameters for the number of frames the note occupies. Then, these note expression and pitch parameters are concatenated with note onsets and offsets (a binary flag the start and end of note, respectively). and scalar note positioning code to provide additional information about note boundaries.
note-length sequence by repeating
B.4 SYNTHESIS GENERATOR
The Synthesis Generator is Generative Adversarial Network (GAN) Goodfellow et al. (2014) whose generator takes the per-note Expression Controls, and produces the DDSP synthesis param- eters. The architecture of the Synthesis Generator is shown in Figure [1 The Synthesis Generator
19

---

Published as a conference paper at ICLR 2022
itself contains two networks: autoregressive RNN, and stack of dilated 1-D convolutions. The autoregressive RNN that generates an fo curve based on a note sequence (c.g., MIDI) and the cor- responding Note Expression Controls (described in Section B.3). The CNN stack then generates an amplitude curve, a harmonic distribution, and noise magnitudes which are passed to the DDSP modules for synthesis. The generated fo curve, amplitude curve, harmonic distribution, and noise magnitudes are all passed to a discriminator.
Jo Generation using the Autoregressive RNN The autoregressive RNN for Jo generation con- sists of single-layer Bi-LSTM and 2-layer GRU that autoregressively samples the note's exact pitch, by predicting fo frequency offset (in units of semitones) with respect to a known fo MIDI note number for the current MIDI note. For example, if at some frame the note specified by the note sequence is A4, which has a MIDI numbe of fo = 69, and the ground truth fo in the audio is f 69.2, the autoregressive RNN is expected predict f - f0r 0.2, indicating that the fo at the current frame should be 0.2 semitones above the fo determined by the current MIDI note, f 69. The autoregressive RNN outputs categorical distribution of pitch offsets in the range of [-0, 1.00] semitones, quantized to 201 bins, where each bin represents 0.01 semitone. The final frequency. fo. imput to DDSP synthesizer is converted from semitone units to Hz using f(n) 440 2 where is the frequency in Hz, and is the integer MIDI note number. Both the single-layer Bi-LSTM and 2-layer GRU in the Synthesis Generator have a hidden dimen- sion of 256. At inference time, the pitch offset is sampled using nucleus sampling (Holtzman al.) 2020) with = 0.95 to avoid sudden unrealistic change in fundamental frequency contour. Sim- ilar approaches for using autoregressive RNNs to sample fo curves has been proposed for speech synthesis and conversion Wang et al. 80186
Morrison et al. 2020).
Generating the Rest of the Synthesis Parameters A stack of dilated I-D convolution layers, a la WaveNet (van den Oord et al., 2016), is used to generate the rest of synthesis parameters, i.e., the base amplitude for the note, af). the harmonic amplitudes, h(t), and the noise magnitudes (!). This network uses the predicted fo(), and concatenated expression control vector, c(l), as conditioning. This network consists of convolutional stacks, with each stack having five layers of I-D convolution with exponentially increasing dilation rate followed by ReLU activation (Maas iet al. 2013) and layer normalization (Ba et al. 2016). For clarity, we refer to this network as the "Dilated CNN."
The conditioning sequence input is mapped to being sent into an autoregressive RNN and dilated convolution network. If trained on the multi- instrument dataset, 64-dimension instrument embedding is concatenated after the linear layer. A dropout of rate 0.5 applied to all GRU units, and during training the input is teacher-forced to avoid overfitting and exposure bias.
vector by linear layer before
The full details of the dilated convolutional network architecture are shown in Table 4
Diseriminator for the Synthesis Generator The architecture of the discriminator used with the Synthesis Generator is shown in Figure 12 and the detailed architecture of each discriminator block is shown in Table 5 The discriminator is motivated by the multi-scale discriminator network in previous works generating waveforms and spectrograms (Kumar et al.] 2019 Lee et al., 2021a). It consists of discriminator networks with same architecture that take input signals with different downsample rate in an effort to learn the features of synthesis parameters at different time resolu- tions. Each discriminator network consists of 4 blocks at each scale, and each block extracts fea- tures from predicted synthesis parameters and the conditioning sequence. For each feature stream in block, two I-D convolutional layers are used with Leaky-ReL.U activation function (Xu et al., 2015), with skip connections and layer normalization (Ba et al. 2016).
Synthesis Generator Losses tion loss and adversarial loss:
The Synthesis Generator is trained by minimizing both reconstruc-
Crecon + aLadv (CCE()) + Lspec) + aLadv:
(11)
hhttps://ww. ottpr//aaaaiiiiooooooooooooooooa www.ootttnmberraaanaaaaaaaaa _entterr_______
Frequencies
20

---

Published as a conference paper at ICLR 2022
Dilated Stack
Kernel Size
Dilation Rate
Stride
Filter Size
Conv ld
3
K Kritters
ReLU + layer norm
Add residual
Convld
3
2
K Filters
ReLU + layer norm
Add residual
Convid
3
4
K Kittees
ReLU + layer norm
Add residual
Convld
3
8
K Futter
ReLU + layer norm
Add residual
Convid
3
16
Kyitters
ReLU + layer norm
Add residual
Dilated CNN
Output Size
Kernel Size
Dilation Rate
Stride
Filter Size
Conditioning Sequence
(1000, 384)
Convld
(1000. 128)
3
128
Dilated Stack
(1000. 128)
3
128
Dilated Stack
(1000, 128)
3
128
Dilated Stack
(1000, 128)
3
128
Dilated Stack
(1000, 128)
3
128
Layer norm
(1000, 128)
Dense
(1000, 126)
126
Table 4: The architecture of dilated convolution network (bottom) used in the Synthesis Generator to generate amplitude, harmonic distribution and noise magnitude. The dilated convolution network uses dialted stack layers defined in the first table below.
Discriminator Block conditioning sequence
Output Size
Kernel Size
Stride
Filter Size
Conditioning Sequence Convid
(T Dz) (Tin/ 256)
256
Add output from Discriminator Block synthesizer parameters
(2,/2,, 256)
Leaky ReLU Add residual and layer norm
(Tnn22, 256) (T., 256)
Discriminator Block
synthesis parameters
Output Size
Kernel Size
Stride
Filter Size
synthesis parameters
(Tins D²)
Convid
(Tin/2, 256)
3
2
256
Leaky ReLU Add residual and layer norm
(Tin/2 256)
(T /2, 256)
Table Details of the discriminator blocks used in the Synthesis Generator.
where Crecon is the reconstruction loss, Ladv is the adversarial loss, and a a hyperparameter that controls the weight of the adversarial loss term. The reconstruction loss, Crecons consists of two pieces: a cross-ent cnosspyn loss, CCE()) on the fo to train the autoregressive RNN, and multi-scale spectral loss, Lypec to train the Dilated CNN.
21

---

Published as a conference paper at ICLR 2022
4x Discriminator Block
Synthesizer Parameters
1D Conv
Leaky ReLU
1D Conv
Add & Norm
Feature Map
Linear
Output
Conditioning Sequence
1D Conv
Leaky ReLU
1D Conv
Add & Norm
Figure 12: The architecture of the discriminator used in the Synthesis Generator.
The cross-entropy loss for the autoregressive RNN defined as
CCE(F)) Σ fue log fus.
(12)
where i is the length of the entire sequence in frames, fo is the ground truth fundamental frequency curve, and fo is the estimated fundamental frequency curve.
The multi-scale spectral loss, Lapecs is used for the reconstruction loss. This loss is used recon- struction in the original DDSP paper (Engel et al., (020a)). The multi-scale spectral loss computes the LI difference between the magnitude spectrogram of the predicted and target audio by comparing the computed spectrograms
number of different FFT sizes. Given the magnitude spectrogram of the predicted audio S, and that of the target audio S, with FFT size i, the multi-scale spectral loss computes the /11 difference between S, and S; well log S, and log
C -11S; --1ll1 + B|| log S; - log Silli,
(13)
Sepec Σ c(i)
Vi E {2048, 1024, 512, 256, 128, 64]. spec
(14)
The adversarial loss, Ladv, used to train the dilated CNN in the Synthesis Generator. This loss com- bines least-squares GAN (LSGAN) (Mao al. 2017) objective and feature matching objective objective (Kumar et al. 2019):
Lads = Liagon + 7h fm-
(15)
Given discriminator network Dr in k-th scale, the LSGAN objective train the Synthesis Generator can be written as
Lisgan Ee Σ (Dx 1|12
(16)
and the LSGAN objective for training the discriminator is given by
min E [[] c) - 11]2 + +|I c) (2), Vk. Dt
(17)
In this work, 3 scales, 2.3 Given the output of i-th feature map from of the 4 layers of the k-th discriminator D the feature map matching objective is calculated as L1 difference between corresponding feature map:
22

---

Published as a conference paper at ICLR 2022
Ground-truth
Without GAN
With GAN
Figure 13: The effect GAN in overcoming the "over-smoothing" problem in harmonic distribution generation. From top to bottom: (top) the ground-truth harmonic distribution of the test-set sample, (middle) the harmonic distribution of the same sample predicted without discriminator used in training, (bottom) the harmonic distribution predicted with discriminator and adversarial training.
fim Eg,c
(s,c)) (8, cill
(18)
N
where N, is the number of units in i-th layer of the feature map.
In training, we use stop gradient between the adversarial loss and the autoregressive RNN. That is, the RNN is trained by the cross-entropy loss only. The Synthesis Generator is trained using sequence length of 1000. The Synthesis
seconds of audio with a frame length of 4ms, resulting Generator is optimized via Adam optimizer in batch size of 16 and a learning rate of 0.0003, with an exponential learning rate decay at a rate of 0.99 per 1000 steps. The discriminator optimized using Adam optimizer in batch size of 16 and learning rate of 0.0001. = 1, 3 1, and 7 = 10 are used for loss coefficients.
B.5 ON THE IMPROVEMENT OF USING GAN FOR THE SYNTIEESIS GENERATOR
Without GAN training, Synthesis Generator will suffer from the "over-smoothing" problem where the model generates uniform and smoothed harmonic distribution and noise magnitudes over the whole note. Figure 1 shows this effect qualitatively. With the introduction of adversarial training. the proposed model overcomes the over-smoothing problem potentially caused by one-to- many mapping.
B.6 EXPRESSION GENERATOR
The Expression Generator creates a set of Expression Controls (see Section B.3 given an input note sequence. The architecture of the Expression Generator is shown in Figure [4 The Expression Generator consists two parts: a single-layer bidirectional GRU extracts context information from input, and two-layer autoregressive GRU generates note expression.
23

---

Published as a conference paper at ICLR 2022
Notes
Pitch
Embedding
Duration
Dense
Autoregressive GRU
Note Expression Controls
Bi-GRU
Instrument Embedding
Expression Generator
Figure 14: The architecture of the Expression Generator, The Expression Generator consists of two parts: single-layer bidirectional GRU extracts context information from input, and two-layer au- toregressive GRU generates note expression. The input to the bi-directional GRU a concatenation of pitch embedding vector, duration feature vector and instrument embedding.
the input to Expression Generator, discrete pitch mapped into feature vector of dimensions via embedding layer, and the scalar duration is mapped into feature vector of 64 dimensions via fully-connected layer. The instrument embedding input is 64 dimension feature vector. The input to the bi-directional GRU is concatenation of pitch embedding vector, duration feature vector and instrument embedding. The bi-directional GRU and auto-regressive GRU all hidden size of 128 and dropout rate of 0.5. Input dropout with rate of 0.5 is applied to the teacher-forced input in training time to avoid over-fitting and exposure bias. The output layer of the Expression Generator two-layer Multi-layer Perceptron (MLP), which consists of fully connected layer with a layer normalization before the ReLU nonlinearity used in Engel et al. (2020a).
Data augmentation is applied in the training of the Expression Generator. For the input note quence, we randomly transpose {-3. 22 - -,,0, 1, 23 semitone(s) and randomly stretch the dura- tion by a factor of (0.9, 0.95, 1.05, 1.1].
The Expression Generator is trained on a sequence length of 64 notes and batch size of 256. Adam optimizer (Kingma & Ba, 2014) used in training with learning rate 0.0001. For training the Expression Generator, we use the mean-square loss (MSE) between the estimated Expression Controls and the ground truth Expression Controls (see Section B.3
B.7 OTHER TRAINING DETAILS
The Expression Generator, Synthesis Generator, and DDSP Inference Module trained separately. The Expression Generator is trained for 5000 steps, the Synthesis Generator is trained for 40000 steps, and the DDSP Inference Module trained for 10000 steps.
C DATASET
The 13 instruments in the URMP dataset are violin, viola, cello, double bass, flute, oboe, clarinet, saxophone, bassoon, trumpet, horn, trombone, tuba. There were originally 14 instruments in URMP dataset, where soprano saxophone and tenor saxophone are regarded as different instrument. In this paper, we regard both the soprano saxophone and tenor saxophone URMP dataset saxophone. The recordings in the dataset have sample rate of 48kHz but we downsampled them to 16kHz to match the sample rate of the DDSP synthesis module. train the Synthesis Generator and DDSP Inference, segmented the recordings into 4 seconds clips with 50% overlap.
In the URMP dataset, the solo recordings are part of ensemble pieces. Splitting the same piece played by different instruments into training and test sets can cause data leakage. Thus, split the dataset based on random shuffle of the recordings, and then moved pieces post-hoe such that the same piece does not appear in both training and test set. We use solo recordings in piece number [3, 9, 11, 21, 24, 25, 33, 38, 39, 40, 41, 43] in URMP as test set, and the rest of distinct solo recordings in URMP as training set, there repeat of solo recordings among different pieces.
24

---

Published as a conference paper at ICLR 2022
D EXPERIMENT DETAILS
D.1 DETAILS OF BASELINE METHODS
We use the orchestral strings pack in Abletor without any manual adjustment synthesize the audio from Ableton. For FluidSynth, we used the FluidR3. _CM. sf2 sound font. We reim- plemented the MIDI2Param system proposed by (Castellon et al. 2020) by following the official implementation on GitHut The MIDI2Params system is only trained in the single instrument set- ting for the listening test experiments as done in the original paper. However, as comparison in multi-instrumeen scenario, trained multi-instrument MIDI2Params model that had an additional instrument embedding concatenated to the model input.
D.2 DETAILS OF THE LISTENING TEST
In total, gathered 960 ratings based 360 pairwise comparisons from 14 participants for our listening test. Participants were presented with two unlabeled audio clips and asked "Which one of the following recordings sound most like person playing it on a real violin?" Participants were asked to wear headphones, and were able to listen to the audio clips as many times as they pleased before submitting their answers. The participants chose their preference using a 5-point Likert-likk scale, with the first option being "Strong preference for Audio Clip I", the middle option indicating "No preference", and the final option being "Strong preference for Audio Clip 2."
The pairs of unlabeled clips were drawn from the set of [Ground-truth, DDSP Inference, MIDI- DDSP, Ableton, MIDI2Params, Fluidsynth). Participants were recruited through Google internal data labeling platform, and were selected based pre-screening pilot study ensure that the participant was able to provide reasonable evaluations audio recordings, In this pilot study, we filtered out individuals who rated FluidSynth examples sounding more like real violin record- ing than the Ground-truth violin recording. Participants were not screened based on their musical background.
A Kruskal-Wallis H test of the ratings showed that there is at least one statistically significant differ- ence between the models: x2) 395.35, p < 0.01. The number of wins for each pair comparison and Wilcoxon signed-rame test for each pair is shown in Table 6
25

---

Published as a conference paper at ICLR 2022
Pairs
wins
ties
losses
p value
Ableton
MIDI2Params
39
1
24
7.6le-5
Ableton
DDSP Inference
11
0
2.98e-27 53
Ableton
FluidSynth
42
0
22
0.0002
Ableton
Ground-truth
14
]
6.62e-26 49
Ableton
MIDI-DDSP
5
0
5.34c-16 59
MIDI2Params
DDSP Inference
8
5.73e-42 56
MIDI2Params
FluidSynth
31
0
33
0.027
MIDI2Params
Ground-truth
56
2.43e-40
MIDI2Params
MIDI-DDSP
8
0
7.16e-28 56
DDSP Inference
FluidSynth
55
0
2.97e-39 9
DDSP Inference
Ground-truth
35
0
0.024 29
DDSP Inference
MIDI-DDSP
44
0
20
6.02e-05
FluidSynth
Ground-truth
4
0
60
1.00e-37
FluidSynth
MIDI-DDSP
9
0
55
7.25e-26
Ground-truth
MIDI-DDSP
44
0
20
0.00018
Table 6: A post-hoc comparison of each pair on their pairwise comparisons with each other, using the Wilcoxon signed-rank test for matched samples ("win" means the first item in the pair is selected) P value less than 0.01 /15 6.67 x 10-4 yields a statistically significant difference. Only two pairs are significantly different (DDSP Inference vs. Ground- truth, MIDI2Params vs. FluidSynth), and are marked with an asterisk (*).
E EXPRESSION ATTRIBUTE CONTROLS
Volume
Vol. Fluc.
Vol. Peak Pos.
Vibrato
Brightness
Attack Noise
violin
.99
.84
.80
.86
.96
.97
viola
.98
.74
.70
.82
.98
.97
cello
.97
.64
.54
.74
.98
.94
double bass
.98
.85
.34
.84
.99
.95
flute
.99
.87
.48
.63
.90
.97
oboe
.97
.79
.72
.91
.87
.97
clarinet
.98
.88
.63
.72
.88
.97
saxophone
.97
.71
.43
.80
.94
96
bassoon
.99
.90
56
.91
.99
.96
trumpet
.97
.88
.55
.73
.92
.92
horn
.97
.88
.41
.64
.94
.95
trombone
.97
.93
.59
.52
.99
.96
tuba
.98
.91
.15
.22
.98
.93
Table 7: The Pearson correlation result of all instruments described in Table 2 The Pearson relation r-values are shown in the table, while p-values are omitted as all entries < 0.0001. We consider Pearson r-values larger than 0.7 (marked on bold) to mean that input controls are strongly correlated with the output. For Volume, Brightness, and Attack Noise, all instruments show strong correlation; for Volume Fluctuation, and Vibrato, the majority of instruments sho strong correlation.
F ACKNOWLEDGEMENT AND OPEN-SOURCE IMAGE ATTRIBUTIONS
We would like to thank Yi Ren for the discussion on Generative Adversarial Nets for audio genera- tion. We would like to thank Shujun Han for providing advice making and arranging figures in the paper. We would also like to thank Mark Cartwright for his feedback on the paper and fruitful discussions about it.
26

---

Published as a conference paper at ICLR 2022
The icons used throughout the paper used under the Creative Commons license via the Noun Project. We gratefully acknowledge the following creators of these images:
Audio by cataicon from the Noun Project.
bassoon by Symbolon from the Noun Project.
Clarinet by Symbolon from the Noun Project.
composer by Pham Duy Phuong Hung from the Noun Project.
Flute by Symbolon from the Noun Project.
Neural Network by Ian Rahmadi Kurniwwan from the Noun Project.
oboe by Symbolon from the Noun Project.
Synthesizer by Jino from the Noun Project.
Violin by Olena Panasovska from the Noun Project.
Violinist by Luis Prado from the Noun Project
27


Reviews:
[{"review_id": "wSM7YK7Qy-b", "reviewer": "Reviewer_FkaH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces a layer of controllable expression parameters for the audio rendering of MIDI files. The contribution is threefold:\n1. The authors introduce heuristics to extract note-wise expression parameters from low-level synthesis parameters. The choice of the controls are relevant to the dataset the authors use (acoustic instruments).\n2. They introduce the synthesis generator, allowing to predict the frame-level synthesis parameters, hence bypassing the DDSP decoder that used to predict them from F0 and loudness contours. The introduction of this model allows a better reconstruction error than the original DDSP. Also, it transfers the manipulation of the sound from the mentioned contours (which are frame-level) to note-level expression controls, more expressive and controllable.\n3. The smart design of the expression controls allows them to be controlled by humans, hence allowing further manipulation of the synthesis. Each control is well defined, normalized between 0 and 1, and coarse enough (note-level vs. frame-level) so that a user can easily set them to their preferred value, enhancing the customization possibilities. Human manipulation of the controls seem to have the desired effect, as shown by the correlation study in Table 2 and further supported by the audio material.\nA few comments:\n- In paragraph 3.3, the authors write that the note expression controls are pooled over the duration of the corresponding note. To my understanding, the controls would rather be unpooled, repeated or upsampled, as there are more frames than notes (very clear in B.4).\n- The paragraph about the dataset doesn't mention that it also contains the MIDI ground truth. This could be made explicit.\n- The paragraph entitled \"Expression Generator\" in 4.2 features what looks like a residue of an unwanted sentence\n- In Appendix B.3, the definition for brightness seems to be lacking a sum over k, and I'm having a hard time trying to understand the multiplication by i rather than k.\n- Some of these expression controls naturally lie in [0;1] (e.g. Volume Peak Position). However, it is not clear how others are normalized (e.g. Vibrato). If normalizing constants are precomputed using feature extraction on the whole dataset, it should be written.\n", "summary_of_the_paper": "This paper presents a controllable rendering engine for MIDI files, based on the DDSP framework. Given F0 and loudness contour, DDSP can estimate the parameters of a harmonic + noise synthesis model, to render a corresponding audio file. Similar to MIDI2Params, which predicts framewise FO and loudness contours from a MIDI file, MIDI DDSP introduces an intermediate hierarchical level, allowing to control some newly introduced \"expression controls\". A mapping from MIDI files to \"expression controls\" is learnt, so that MIDI files can be automatically rendered. However, because these note wise controls are explicit, they also allow human manipulation of the performance rendering.\nInfluence of the expression controls is assessed with a correlation study, showing that the human manipulation has the expected effect on the generated performance. This quantitative evaluation is further confirmed by convincing audio examples.", "summary_of_the_review": "Overall, this works is very well written. Everything should appear pretty clear to anyone familiar with the DDSP framework. The principle is simple yet extremely effective, as it improves reconstruction quality and parameters estimation compared to the state of the art. This new control layer represents a lot of added value compared to MIDI2Params, which was already a great addition to the original DDSP framework. In addition to the paper, the website provides very convincing audio examples which further assess the quality of this work.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "kQ8am1nsc8", "reviewer": "Reviewer_Z9iV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, so I only have few minor questions.\n\nWhile reading the paper, I was wondering the three sub-modules are trained simultaneously. If not, can other dataset be used for training each sub-module to enhance robustness or the performance?\n\nFor synthesis generator module, I couldn't find some ablation test about the loss function. \n\nThe authors argue that the method can be easily extended when multi-instrument transcription model is ready. However, I see there exist many challenges (e.g. the model relies on CREPE in DDSP synthesis part, also the type of an instrument set between transcription part and MIDI-DDSP should be matched.), the tone can be lowered.\n\nIn page 3, \"Realistic Note Synthesis\", \"a\" is missing in a word 'concatenative'.\n", "summary_of_the_paper": "This paper proposes a music performance modeling network using three sub-modules which are expression generator, synthesis generator, and DDSP inference. The idea of using these three sub-modules to create three-level performance modeling (the three-level control values are notes, performance features, and synthesis parameters) is interesting and the result shows that the proposed model can give more ability of control while not harming the overall audio generation performance. It seems the proposed method that having three trainable networks with two hand-crafted methods (actually note detection is not the part of the contribution of this paper, so I didn't count it) gives full pipeline of hierarchical modeling and this is a good research contribution.", "summary_of_the_review": "The paper opens a more user controllability in music performance modeling and this is a nice contribution. Also, the idea of having rule-based method with trainable networks in a series of a chain seems useful in many other applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bVn-0fN5wN5", "reviewer": "Reviewer_6Ata", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper develops a parametric model of musical audio that leverages insight into the music domain to construct meaningful featurizations of music, together with models trained to invert these featurizations, to construct a hierarchical autoencoder with meaningful semantic features that can be used to control the music generation process.\n\nThe technical contributions of this paper consist of a semantic \"expression\" featurization of DDSP parameters (Section 3.2) and two inverse models: (1) a synthesis generator for converting the expression features back to DDSP synthesis parameters and (2) an expression generator for converting MIDI data back to expression features. The features and models are well-described at a high level in the main paper, and the appendix provides a thorough & precise description of the remaining details. The empirical study seems convincing, with comparisons to both academic prior work (MIDIParams) and commercial software (Ableton, FluidSynth). And the demos sound qualitatively good to my ear.\n\nThe paper thoroughly relates this work to related work in the field, making many thoughtful connections to both the symbolic & audio generative modeling literature. These connections both help situate this work in the broader literature, and also provide some welcome scaffolding for thinking more broadly about the field of generative music modeling. I also appreciate the connections to the signal processing literature.\n\n\nOne possible criticism of this paper is that, because its focus is on applying machine learning rather than in the development of machine learning techniques, perhaps it would be better suited for a music domain conference (e.g., ISMIR). I strongly support its publication at ICLR: the paper does a thorough job situating itself in the machine learning literature, making it accessible and appealing to the broader machine learning audience.\n\n\nOne question I have is about the choice of 3 stage modeling (notes -> expression -> synthesis -> audio) as opposed to the 2 stage process used by MIDI2Params (notes -> synthesis -> audio). I appreciate that injecting semantically meaningful performance features provides an additional level of control over the generation process. Does this additional stage also account for the performance improvement of MIDI-DDSP over MIDI2Params? Or is this improvement more attributable to better/bigger models? \n\nAlso, related to the previous point, I am a little confused by the MIDI2Params comparisons in Table 1. I am under the impression that the expression step is an innovation of this work (Section 3.2). How are you able to make these breakdown comparisons with MIDI2Params for these substages notes -> expression and expression -> synthesis. Have I misunderstood MIDI2Params?", "summary_of_the_paper": "This paper extends the DDSP generative modeling approach (Engel et al., 2020) to higher levels of abstraction for high-level MIDI -> audio synthesis with intermediate levels of semantic control over the generation process.", "summary_of_the_review": "This is a well-written, creative paper. The models are interesting. The figures are excellent. The connections to related work are extensive and thoughtful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed MIDI-DDSP, a structured hierarchical generative model which offers both detailed expressive controls (as in traditional synthesizers) as well as the realistic audio quality (as in black-box neural audio synthesis). Overall the reviews are very positive. All the reviewers unanimously agree that the paper is very well-written and presented a very convincing model and a meaningful step-up from the earlier work of DDSP. The authors also presented a well-documented website for the project and promised to release the source code. The reviewers raised some clarifying questions and minor corrections which the authors addressed during the response. Therefore, I vote for accept.
Decision: Accept (Oral)