MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

1 INTRODUCTION . Generative models are most useful to creators if they can generate realistic outputs , afford many avenues for control , and easily fit into existing creative workflows ( Huang et al. , 2020 ) . Deep generative models are expressive function approximators , capable of generating realistic samples in many domains ( Ramesh et al. , 2021 ; Brown et al. , 2020 ; van den Oord et al. , 2016 ) , but often at the cost of interactivity , restricting users to rigid black-box input-output pairings without interpretable access to the internals of the network . In contrast , structured models chain several stages of interpretable intermediate representations with expressive networks , while still allowing users to interact throughout the hierarchy . For example , these techniques have been especially effective in computer vision and speech , where systems are optimized for both realism and control ( Lee et al. , 2021b ; Chan et al. , 2019 ; Zhang et al. , 2019 ; Wang et al. , 2018a ; Morrison et al. , 2020 ; Ren et al. , 2020 ) . For music generation , despite recent progress , current tools still fall short of this ideal ( Figure 1 , right ) . Deep networks can either generate realistic full-band audio ( Dhariwal et al. , 2020 ) or provide detailed controls of attributes such as pitch , dynamics , and timbre ( Défossez et al. , 2018 ; Engel et al. , 2019 ; 2020a ; Hawthorne et al. , 2019 ; Wang & Yang , 2019 ) but not both . Many existing workflows use the MIDI specification ( Association et al. , 1996 ) to Conventional DSP synthesizers ( Chowning , 1973 ; Roads , 1988 ) provide extensive control but make it difficult to generate realistic instrument timbre , while concatenative samplers ( Schwarz , 2007 ) play back high-fidelity recordings of isolated musical notes , but require manually stitching together performances with limited control over expression and continuity . In this paper , we propose MIDI-DDSP , a hierarchical generative model of musical performance to provide both realism and control ( Figure 1 , left ) . Similar to conventional synthesizers and samplers that use the MIDI standard ( Association et al. , 1996 ) , MIDI-DDSP converts note timing , pitch , and expression information into fine-grained parameter control of DDSP synthesizer modules . We take inspiration from the hierarchical structure underlying the process of creating music . A composer writes a piece as a series of notes . A performer interprets these notes through a myriad of nuanced , sub-second choices about articulation , dynamics , and expression . These expressive gestures are realized as audio through the short-time pitch and timbre changes of the physical vibration of the instrument . MIDI-DDSP is built on a similar 3-level hierarchy ( notes , performance , synthesis ) with interpretable representations at each level . While the efficient DDSP synthesis representation ( low-level ) allows for high-fidelity audio synthesis ( Engel et al. , 2020a ) , users can also control the notes to be played ( high-level ) , and the expression with which they are performed ( mid-level ) . A qualitative example of this is shown in Figure 2 , where a given performance on violin is manipulated at all three levels ( notes , expression , synthesis parameters ) to create a new realistic yet personalized performance . As seen in Figure 1 ( left ) , MIDI-DDSP can be viewed similarly to a multi-level autoencoder . The hierarchy has three separately trainable modules ( DDSP Inference , Synthesis Generator , Expression Generator ) and three fixed functions/heuristics ( DDSP Synthesis , Feature Extraction , Note Detection ) . These modules enable MIDI-DDSP to conditionally generate at any level of the hierarchy , providing creative assistance by filling in the details of a performance , synthesizing audio for new note sequences , or even fully automating music generation when paired with a separate note generating model . It is important to note that the system relies on pitch detection and note detection , so is currently limited to training on recordings of single monophonic instruments . This approach has the potential to be extend to polyphonic recordings via multi-instrument transcription ( Hawthorne et al. , 2021 ; Engel et al. , 2020b ; Bittner et al. , 2017 ) and multi-pitch tracking , which is an exciting avenue to explore for future work . Finally , we also show that each stage can be made conditional on instrument identity , training on 13 separate instruments with a single model . For clarity , we summarize the core contributions of this work : • We propose MIDI-DDSP , a 3-level hierarchical generative model of music ( notes , performance , synthesis ) , and train a single model capable of realistic audio synthesis for 13 different instruments . ( Section 3 ) • Expression Attributes : We introduce heuristics to extract mid-level per-note expression attributes from low-level synthesis parameters . ( Figure 4 ) • User Control : Quantitative studies confirm that manipulating the expression attributes creates a corresponding effect in the synthesizer parameters , and we qualitatively demonstrate the detailed control that is available to users manipulating all three levels of the hierarchy . ( Table 2 and Figure 2 ) • Assistive Generation : Reconstruction experiments show that MIDI-DDSP can make assistive predictions at each level of the hierarchy , accurately resynthesizing audio , predicting synthesis parameters from note-wise expression attributes , and auto-regressively predicting note-wise expression attributes from a note sequence . ( Tables 1a , 1b , 1c ) • Realistic Note Synthesis : An extensive listening study finds that MIDI-DDSP can synthesize audio from new note sequences ( not seen during training ) with higher realism than both comparable neural approaches and professional concatenative sampler software . ( Figure 5 ) • Automatic Music Generation : We demonstrate that pairing MIDI-DDSP with a pretrained note generation model enables full-stack automatic music generation . As an example , we use Coconet ( Huang et al. , 2017 ) to generate and synthesize novel 4-part Bach chorales for a variety of instruments . ( Figure 6 ) Audio samples of all results and figures are provided in the online supplement1 . We highly recommend readers to access the online supplement to the paper . 2 RELATED WORK . Note Synthesis . Existing neural synthesis models allow either high-level manipulation of note pitch , velocity , and timing ( Hawthorne et al. , 2019 ; Kim et al. , 2019 ; Wang & Yang , 2019 ; Manzelli et al. , 2018 ) , or low-level synthesis parameters ( Jonason et al. , 2020 ; Castellon et al. , 2020 ; Blaauw & Bonada , 2017 ) . MIDI-DDSP connects these two approaches by enabling both high-level note controls and low-level synthesis manipulation in a single system . 1https : //midi-ddsp.github.io/ Most related to this work is MIDI2Params ( Castellon et al. , 2020 ) , a hierarchical model that autoregressively predicts frame-wise pitch and loudness contours to drive the original DDSP autoencoder ( Engel et al. , 2020a ) . MIDI-DDSP builds on this work by adding an additional level of hierarchy for the note expression , training a new more accurate DDSP base model , and explicitly modeling the synthesizer coefficients output by that model , rather than the pitch and loudness inputs to the model . We extensively compare to our reimplementation of MIDI2Params as a baseline throughout the paper . Hierarchical Audio Modelling . Audio waveforms have dependencies over timescales spanning several orders of magnitude , lending themselves to hierarchical modeling . For example , Dieleman et al . ( 2018 ) and Dhariwal et al . ( 2020 ) both choose to encode audio as discrete latent codes at different time resolutions , and apply autoregressive models as priors over those codes . MIDI-DDSP applies a similar approach in spirit , but constructs a hierarchy based on semantic musical structure ( note , performance , synthesis ) , allowing interpretable manipulation by users . Expressive Performance Analysis and Synthesis . Many prior systems pair analysis and synthesis functions to capture expressive performance characteristics ( Canazza et al. , 2004 ; Yang et al. , 2016 ; Shih et al. , 2017 ) . Such methods often use heuristic functions to generate parameters for driving synthesizers or selecting and modifying sample units . MIDI-DDSP similarly uses feature extraction , but each level is paired with a differentiable neural network function that directly learns the mapping to expression and synthesis controls for more realistic audio synthesis . 3 MODEL ARCHITECTURE . 3.1 DDSP SYNTHESIS AND INFERENCE . Differentiable Digital Signal Processing ( DDSP ) ( Engel et al. , 2020a ) enables differentiable audio synthesis by using a harmonic plus noise model ( Serra & Smith , 1990 ) . Full details are provided in Appendix B.1 . Briefly , an oscillator bank synthesizes a harmonic signal from a fundamental frequency f0 ( t ) , a base amplitude a ( t ) , and a distribution over harmonic amplitudes h ( t ) , where the dimensionality of h is the number of harmonics . The noise signal is generated by filtering uniform noise with linearly spaced filter banks , where η ( t ) represents the magnitude of noise output from each filter in time . In this study , we use 60 harmonics and 65 noise filter banks , giving 127 total synthesis parameters each time frame ( s ( t ) = ( f0 ( t ) , a ( t ) , h ( t ) , η ( t ) ) ) . The final audio is the addition of harmonic and noise signals . Since the synthesis process is differentiable , Engel et al . ( 2020a ) demonstrate that it is possible to train a neural network to predict the other synthesis parameters given f0 ( t ) and the loudness of the audio , and optimize a multi-scale spectral loss ( Wang et al. , 2019 ; Engel et al. , 2020a ) of the resynthesized audio ( Figure 3 left ) . f0 ( t ) is extracted by a pre-trained CREPE model ( Kim et al. , 2018 ) , and the loudness is extracted via an A-weighting of the power spectrum ( Hantrakul et al. , 2019 ; McCurdy , 1936 ) . We extend this work for our DDSP Inference module , by providing and additional input features a log-scale Mel-spectrogram of the audio , that produces higher quality resynthesis ( Table 1a ) . Full architectural details are provided in Appendix B.2 . 3.2 EXPRESSION CONTROLS . We aim to model aspects of expressive performance with a continuous variable . For example , this enables a performer to choose how loud the note should be performed , or how much vibrato to apply . We define six expression controls ( detailed in Appendix B.3 ) , scaled within [ 0 , 1 ] . These are extracted from synthesis parameters s ( t ) and applied within the ith note , ni ( t ) , in a note sequence : Volume : Controls the volume of a note , extracted by taken average amplitude over a note . Volume fluctuation : Determines the magnitude of a volume change across a note . Used with the volume peak position , below , this can make a note crescendo or decrescendo . This is extracted by calculating the standard deviation of the amplitude over a note . Volume peak position : Controls where , over the duration of a note , the peak volume occurs . Zero value corresponds to decrescendo notes , whereas one corresponds to crescendo notes . The volume peak position is extracted by calculating the relative position of maximum amplitude in the note . Vibrato : Controls the extent of the vibrato of a note . Vibrato is a musical technique defined by pulsating the pitch of a note . Vibrato is extracted by applying Discrete Fourier Transform ( DFT ) on the fundamental frequency f0 ( t ) in a note and take the peak amplitude . Brightness : Controls the timbre of a note where a higher value corresponds to larger high-frequency harmonic . Brightness is extracted by calculating the average harmonic centroid of a note . Attack Noise : Controls how much noise occurs at the start of the note ( the attack ) , e.g. , the fluctuation of string and bow . At certain settings , this determines whether two notes sound consecutively or separately . The attack noise is extracted by taking a note ’ s average noise magnitude in the first ten frames ( 40ms ) .

Summary: This paper presents a controllable rendering engine for MIDI files, based on the DDSP framework. Given F0 and loudness contour, DDSP can estimate the parameters of a harmonic + noise synthesis model, to render a corresponding audio file. Similar to MIDI2Params, which predicts framewise FO and loudness contours from a MIDI file, MIDI DDSP introduces an intermediate hierarchical level, allowing to control some newly introduced "expression controls". A mapping from MIDI files to "expression controls" is learnt, so that MIDI files can be automatically rendered. However, because these note wise controls are explicit, they also allow human manipulation of the performance rendering. Influence of the expression controls is assessed with a correlation study, showing that the human manipulation has the expected effect on the generated performance. This quantitative evaluation is further confirmed by convincing audio examples.