Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

1 INTRODUCTION . Since the beginning of the recent wave of deep learning research , there have been many attempts to create generative models of expressive musical audio de novo . These models would ideally generate audio that is both musically and sonically realistic to the point of being indistinguishable to a listener from music composed and performed by humans . However , modeling music has proven extremely difficult due to dependencies across the wide range of timescales that give rise to the characteristics of pitch and timbre ( short-term ) as well as those of rhythm ( medium-term ) and song structure ( long-term ) . On the other hand , much of music has a large hierarchy of discrete structure embedded in its generative process : a composer creates songs , sections , and notes , and a performer realizes those notes with discrete events on their instrument , creating sound . The division between notes and sound is in many ways analogous to the division between symbolic language and utterances in speech . The WaveNet model by van den Oord et al . ( 2016 ) may be the first breakthrough in generating musical audio directly with a neural network . Using an autoregressive architecture , the authors trained a model on audio from piano performances that could then generate new piano audio sample-bysample . However , as opposed to their highly convincing speech examples , which were conditioned on linguistic features , the authors lacked a conditioning signal for their piano model . The result was audio that sounded very realistic at very short time scales ( 1 or 2 seconds ) , but that veered off into chaos beyond that . Dieleman et al . ( 2018 ) made great strides towards providing longer term structure to WaveNet synthesis by implicitly modeling the discrete musical structure described above . This was achieved by training a hierarchy of VQ-VAE models at multiple time-scales , ending with a WaveNet decoder to generate piano audio as waveforms . While the results are impressive in their ability to capture long-term structure directly from audio waveforms , the resulting sound suffers from various artifacts at the fine-scale not present in the unconditional WaveNet , clearly distinguishing it from real musical audio . Also , while the model learns a version of discrete structure from the audio , it is not directly reflective of the underlying generative process and thus not interpretable or manipulable by a musician or user . Manzelli et al . ( 2018 ) propose a model that uses a WaveNet to generate solo cello music conditioned on MIDI notation . This overcomes the inability to manipulate the generated sequence . However , their model requires a large training corpus of labeled audio because they do not train a transcription model , and it is limited to monophonic sequences . In this work , we seek to explicitly factorize the problem informed by our prior understanding of the generative process of performer and instrument : P ( audio ) = Enotes [ P ( audio|notes ) ] ( 1 ) which can be thought of as a generative model with a discrete latent code of musical notes . Since the latent representation is discrete , and the scale of the problem is too large to jointly train , we split the model into three separately trained modules that are each state-of-the-art in their respective domains : 1 . Encoder , P ( notes|audio ) : An Onsets and Frames ( Hawthorne et al. , 2018 ) transcription model to produce a symbolic representation ( MIDI ) from raw audio . 2 . Prior , P ( notes ) : A self-attention-based music language model ( Huang et al. , 2018 ) to generate new performances in MIDI format based on those transcribed in ( 1 ) . 3 . Decoder , P ( audio|notes ) : A WaveNet ( van den Oord et al. , 2016 ) synthesis model to generate audio of the performances conditioned on MIDI generated in ( 2 ) . We call this process Wave2Midi2Wave . One hindrance to training such a stack of models is the lack of large-scale annotated datasets like those that exist for images . We overcome this barrier by curating and publicly releasing alongside this work a piano performance dataset containing well-aligned audio and symbolic performances an order of magnitude larger than the previous benchmarks . In addition to the high quality of the samples our method produces ( see https : //goo.gl/ magenta/maestro-examples ) , training a suite of models according to the natural musician/instrument division has a number of other advantages . First , the intermediate representation used is more suitable for human interpretation and manipulation . Similarly , factorizing the model in this way provides better modularity : it is easy to independently swap out different performance and instrument models . Using an explicit performance representation with modern language models also allows us to model structure at much larger time scales , up to a minute or so of music . Finally , we can take advantage of the large amount of prior work in the areas of symbolic music generation and conditional audio generation . And by using a state-of-the-art music transcription model , we can make use of the same wealth of unlabeled audio recordings previously only usable for training end-to-end models by transcribing unlabeled audio recordings and feeding them into the rest of our model . 2 CONTRIBUTIONS OF THIS PAPER . Our contributions are as follows : 1 . We combine a transcription model , a language model , and a MIDI-conditioned WaveNet model to produce a factorized approach to musical audio modeling capable of generating about one minute of coherent piano music . 2 . We provide a new dataset of piano performance recordings and aligned MIDI , an order of magnitude larger than previous datasets . 3 . Using an existing transcription model architecture trained on our new dataset , we achieve state-of-the-art results on a piano transcription benchmark . 3 DATASET . We partnered with organizers of the International Piano-e-Competition1 for the raw data used in this dataset . During each installment of the competition , virtuoso pianists perform on Yamaha Disklaviers which , in addition to being concert-quality acoustic grand pianos , utilize an integrated high-precision MIDI capture and playback system . Recorded MIDI data is of sufficient fidelity to allow the audition stage of the competition to be judged remotely by listening to contestant performances reproduced over the wire on another Disklavier instrument . The dataset introduced in this paper , which we name MAESTRO ( “ MIDI and Audio Edited for Synchronous TRacks and Organization ” ) , contains over a week of paired audio and MIDI recordings from nine years of International Piano-e-Competition events.2 The MIDI data includes key strike velocities and sustain pedal positions . Audio and MIDI files are aligned with ≈3 ms accuracy and sliced to individual musical pieces , which are annotated with composer , title , and year of performance . Uncompressed audio is of CD quality or higher ( 44.1–48 kHz 16-bit PCM stereo ) . A train/validation/test split configuration is also proposed , so that the same composition , even if performed by multiple contestants , does not appear in multiple subsets . Repertoire is mostly classical , including composers from the 17th to early 20th century . Table 1 contains aggregate statistics of the MAESTRO dataset . We make the new dataset ( MIDI , audio , metadata , and train/validation/test split configuration ) available at https : //g.co/magenta/maestro-datasetunder a Creative Commons Attribution Non-Commercial Share-Alike 4.0 license . 1http : //piano-e-competition.com 2All results in this paper are from the v1.0.0 release of the dataset . Several datasets of paired piano audio and MIDI have been published previously and have enabled significant advances in automatic piano transcription and related topics . We are making MAESTRO available because we believe it provides several advantages over existing datasets . Most significantly , as evident from table 2 , MAESTRO is around an order of magnitude larger . Existing datasets also have different properties than MAESTRO that affect model training : MusicNet ( Thickstun et al. , 2017 ) contains recordings of human performances , but separatelysourced scores . As discussed in Hawthorne et al . ( 2018 ) , the alignment between audio and score is not fully accurate . One advantage of MusicNet is that it contains instruments other than piano ( not counted in table 2 ) and a wider variety of recording environments . MAPS ( Emiya et al. , 2010 ) contains Disklavier recordings and synthesized audio created from MIDI files that were originally entered via sequencer . As such , the “ performances ” are not as natural as the MAESTRO performances captured from live performances . In addition , synthesized audio makes up a large fraction of the MAPS dataset . MAPS also contains syntheses and recordings of individual notes and chords , not counted in table 2 . Saarland Music Data ( SMD ) ( Müller et al. , 2011 ) is similar to MAESTRO in that it contains recordings and aligned MIDI of human performances on a Disklavier , but is 30 times smaller . 3.1 ALIGNMENT . Our goal in processing the data from International Piano-e-Competition was to produce pairs of audio and MIDI files time-aligned to represent the same musical events . The data we received from the organizers was a combination of MIDI files recorded by Disklaviers themselves and WAV audio captured with conventional recording equipment . However , because the recording streams were independent , they differed widely in start times and durations , and they were also subject to jitter . Due to the large volume of content , we developed an automated process for aligning , slicing , and time-warping provided audio and MIDI to ensure a precise match between the two . Our approach is based on globally minimizing the distance between CQT frames from the real audio and synthesized MIDI ( using FluidSynth3 ) . Obtaining a highly accurate alignment is non-trivial , and we provide full details in the appendix . 3.2 DATASET SPLITTING . For all experiments in this paper , we use a single train/validation/test split designed to satisfy the following criteria : • No composition should appear in more than one split . • Train/validation/test should make up roughly 80/10/10 percent of the dataset ( in time ) , respectively . These proportions should be true globally and also within each composer . Maintaining these proportions is not always possible because some composers have too few compositions in the dataset . • The validation and test splits should contain a variety of compositions . Extremely popular compositions performed by many performers should be placed in the training split . For comparison with our results , we recommend using the splits which we have provided . We do not necessarily expect these splits to be suitable for all purposes ; future researchers are free to use alternate experimental methodologies . 3http : //www.fluidsynth.org/ 4 PIANO TRANSCRIPTION . The large MAESTRO dataset enables training an automatic piano music transcription model that achieves a new state of the art . We base our model on Onsets and Frames , with several modifications informed by a coarse hyperparameter search using the validation split . For full details of the base model architecture and training procedure , refer to Hawthorne et al . ( 2018 ) . One important modification was adding an offset detection head , inspired by Kelz et al . ( 2018 ) . The offset head feeds into the frame detector but is not directly used during decoding . The offset labels are defined to be the 32ms following the end of each note . We also increased the size of the bidirectional LSTM layers from 128 to 256 units , changed the number of filters in the convolutional layers from 32/32/64 to 48/48/96 , and increased the units in the fully connected layer from 512 to 768 . We also stopped gradient propagation into the onset subnetwork from the frame network , disabled weighted frame loss , and switched to HTK frequency spacing ( Young et al. , 2006 ) for the mel-frequency spectrogram input . In general , we found that the best ways to get higher performance with the larger dataset were to make the model larger and simpler . The final important change we made was to start using audio augmentation during training using an approach similar to the one described in McFee et al . ( 2017 ) . During training , every input sample was modified using random parameters for the SoX 4 audio tool using pysox ( Bittner et al. , 2016 ) . The parameters , ranges , and random sampling methods are described in table 3 . We found that this was particularly important when evaluating on the MAPS dataset , likely because the audio augmentation made the model more robust to differences in recording environment and piano qualities . The differences in training results are summarized in table 6 . When evaluating against the MAESTRO dataset , we found that audio augmentation made results slightly worse , so results in table 5 are presented without audio augmentation . After training on the MAESTRO training split for 670k steps , we achieved state of the art results described in table 4 for the MAPS dataset . We also present our results on the train , validation , and test splits of the MAESTRO dataset as a new baseline score in table 5 . Note that for calculating the scores of the train split , we use the full duration of the files without splitting them into 20-second chunks as is done during training . In sections 5 and 6 , we demonstrate how using this transcription model enables training language and synthesis models on a large set of unlabeled piano data . To do this , we transcribe the audio in the MAESTRO training set , although in theory any large set of unlabeled piano music would work . We 4http : //sox.sourceforge.net/ call this new , transcribed version of the training set MAESTRO-T . While it is true that the audio transcribed for MAESTRO-T was also used to train the transcription model , table 5 shows that the model performance is not significantly different between the training split and the validation or test splits , and we needed the larger split to enable training the other models .

Summary: The paper addresses the challenge of using neural networks to generate original and expressive piano music.  The available techniques today for audio or music generation are not able to sufficient handle the many levels at which music needs to modeled.  The result is that while individual music sounds (or notes) can be generated at one level using tools like WaveNet, they don't come together to create a coherent work of music at the higher level.  The paper proposes to address this problem by imposing a MIDI representation (piano roll) in the neural modeling of music audio that serves as an intermediate (and interpretable) representation between the analysis (music audio -> MIDI) and synthesis (MIDI -> music audio) in the pipeline of piano music generation.  In order to develop and validate the proposed learning architecture, the authors have created a large data set of aligned piano music (raw audio along with MIDI representation).  Using this data set for training, validation and test, the paper reports on listening tests that showed slightly less favorable results for the generated music.  A few questions and comments are as follows.  MIDI itself is a rich language with ability to drive the generation of music using rich sets of customizable sound fonts.  Given this, it is not clear that it is necessary to reproduce this function using neural network generation of sounds.  The further limitation of the proposed approach seems to be the challenge of decoding raw music audio with chords, multiple overlayed notes or multiple tracks.  MIDI as a representation can support multiple tracks, so it is not necessarily the bottleneck.  How much does the data augmentation (audio augmentation) help?