Encoding Musical Style with Transformer Autoencoders

1 INTRODUCTION . There has been significant progress in generative modeling , particularly with respect to creative applications such as art and music ( Oord et al. , 2016 ; Engel et al. , 2017b ; Ha & Eck , 2017 ; Huang et al. , 2019a ; Payne , 2019 ) . As the number of generative applications increase , it becomes increasingly important to consider how users can interact with such systems , particularly when the generative model functions as a tool in their creative process ( Engel et al. , 2017a ; Gillick et al. , 2019 ) To this end , we consider how one can learn high-level controls over the global structure of a generated sample . We focus on symbolic music generation , where Music Transformer ( Huang et al. , 2019b ) is the current state-of-the-art in generating high-quality samples that span over a minute in length . The challenge in controllable sequence generation is the fact that Transformers ( Vaswani et al. , 2017 ) and their variants excel as language models or in sequence-to-sequence tasks such as translation , but it is less clear as to how they can : ( 1 ) learn and ( 2 ) incorporate global conditioning information at inference time . This contrasts with traditional generative models for images such as the variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) or generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) which typically incorprate global conditioning as part of their training procedure ( Sohn et al. , 2015 ; Sønderby et al. , 2016 ; Isola et al. , 2017 ; Van den Oord et al. , 2016 ) . In this work , we introduce the Transformer autoencoder , where we aggregate encodings across time to obtain a holistic representation of the performance style . We show that this learned global representation can be incorporated with other forms of structural conditioning in two ways . First , we show that given a performance , our model can generate performances that are similar in style to the provided input . Then , we explore different methods to combine melody and performance representations to harmonize a melody in the style of the given performance . In both cases , we show that combining both global and fine-scale encodings of the musical performance allows us to gain better control of generation , separately manipulating both the style and melody of the resulting sample . Empirically , we evaluate our model on two datasets : the publicly-available MAESTRO ( Hawthorne et al. , 2019 ) dataset , and an internal dataset of piano performances transcribed from 10,000+ hours of audio ( Anonymous for review ) . We find that the Transformer autoencoder is able to generate not only performances that sound similar to the input , but also accompaniments of melodies that follow a given style , as shown through both quantitative and qualitative experiments as well as a user listening study . In particular , we demonstrate that our model is capable of adapting to a particular musical style even in the case where we have one single input performance . 2 PRELIMINARIES . 2.1 DATA REPRESENTATION FOR MUSIC GENERATION . The MAESTRO ( Hawthorne et al. , 2019 ) dataset consists of over 1,100 classical piano performances , where each piece is represented as a MIDI file . The internal performance dataset consists of over 10,000 hours of piano performances transcribed from audio ( Anonymous for review ) . In both cases , we represent music as a sequence of discrete tokens , effectively formulating the generation task as a language modeling problem . The performances are encoded using the vocabulary as described in ( Oore et al. , 2018 ) , which captures expressive dynamics and timing . This performance encoding vocabulary consists of 128 note on events , 128 note off events , 100 time shift events representing time shifts in 10ms increments from 10ms to 1s , and 32 quantized velocity bins representing the velocity at which the 128 note on events were played . 2.2 MUSIC TRANSFORMER . We build our Transformer autoencoder from Music Transformer , a state-of-the-art generative model that is capable of generating music with long-term coherence ( Huang et al. , 2019b ) . While the original Transformer uses a self-attention mechanism that operates over absolute positional encodings of each token in a given sequence ( Vaswani et al. , 2017 ) , Music Transformer replaces this with relative attention ( Shaw et al. , 2018 ) , which allows the model to keep better track of regularity based on event orderings and periodicity in the performance . Huang et al . ( 2019b ) propose a novel algorithm for implementing relative self-attention that is significantly more memory-efficient , enabling the model to generate musical sequences over a minute in length . For more details regarding the self-attention mechanism and Transformers , we refer the reader to ( Vaswani et al. , 2017 ; Parmar et al. , 2018 ) . 3 CONDITIONAL GENERATION WITH THE TRANSFORMER AUTOENCODER . 3.1 MODEL ARCHITECTURE . We leverage the standard encoder and decoder stacks of the Transformer as a foundation for our model , with minor modifications that we outline below . Transformer Encoder : For both the performance and melody encoder networks , we use the Transformer ’ s stack of 6 layers which are each comprised of a : ( 1 ) multi-head relative attention mechanism ; and a ( 2 ) position-wise fully-connected feed-forward network . The performance encoder takes as input the event-based performance encoding of an input performance , while the melody encoder learns an encoding of the melody which has been extracted from the input performance . Depending on the music generation task , which we elaborate upon in Section 3.2 , the encoder output ( s ) are fed into the Transformer decoder . Figure 1 describes the way in which the encoder and decoder networks are composed together . Transformer Decoder : The decoder shares the same structure as the encoder network , but with an additional multi-head attention layer over the encoder outputs . At each step of generation , the decoder takes in the output of the encoder , as well as each new token that was previously generated . The model is trained end-to-end with maximum likelihood . That is , for a given sequence x of length n , we maximize log pθ ( x ) = ∑n i=1 log pθ ( xi|x < i ) with respect to the model parameters θ . 3.2 CONDITIONING MECHANISM . Performance Conditioning and Bottleneck For this task , we aim to generate samples that sound “ similar ” to a conditioning input performance . We incorporate a bottleneck in the output of the Transformer encoder in order to prevent the model from simply memorizing the input ( Baldi , 2012 ) . Thus , as shown in Figure 1 , we mean-aggregate the performance embedding across the time dimension in order to learn a global representation of style . This mean-performance embedding is then fed into the autoregressive decoder , where the decoder attends to this global representation in order to predict the appropriate target . Although this bottleneck may be undesirable in sequence transduction tasks where the input and output sequences differ ( e.g . translation ) , we find that it works well in our setting where we require the generated samples to be similar in style to the input sequence . Melody & Performance Conditioning : Next , we synthesize any given melody in the style of a different performance . Although the setup shares similarities to that of the melody conditioning problem in ( Huang et al. , 2019b ) , we note that we also provide a conditioning performance signal , which makes the generation task more challenging . During training , we follow an internal procedure to extract melodies from performances in the training set , quantize the melody to a 100ms grid , and encode it as a sequence of tokens that uses a different vocabulary than the performance representation . We then use two distinct Transformer encoders ( each with the same architecture ) as in Section 3.1 to separately encode the melody and performance inputs . The melody and performance embeddings are combined to use as input to the decoder . We explore various ways of combining the intermediate representations : ( 1 ) sum , where we add the performance and melody embeddings together ; ( 2 ) concatenate , where we concatenate the two embeddings separated with a stop token ; and ( 3 ) tile , where we tile the performance embedding across every dimension of time in the melody encoding . In all three cases , we work with the meanaggregated representation of the input performance . We find that different approaches work better than others on some dataets , a point which we elaborate upon in Section 5 . Input Perturbation In order to encourage the encoded performance representations to generalize across various melodies , keys , and tempos , we draw inspiration from the denoising autoencoder ( Vincent et al. , 2008 ) as a means to regularize the model . For every target performance from which we extract the input melody , we provide the model with a perturbed version of the input performance as the conditioning signal . We allow this “ noisy ” performance to vary across two axes of variation : ( 1 ) pitch , where we artificially shift the overall pitch either down or up by 6 semitones ; and ( 2 ) time , where we stretch the timing of the performance by at most 5 % . In our experiments , we find that this augmentation procedure leads to samples that sound more pleasing ( Oore et al. , 2018 ) . We provide further details on the augmentation procedure in Appendix A . 4 SIMILARITY EVALUATION ON PERFORMANCE FEATURES . Although a variety of different metrics have been proposed to quantify both the quality ( Engel et al. , 2019 ) and similarity of musical performances relative to one another ( Yang & Lerch , 2018 ; Hung et al. , 2019 ) , the development of a proper metric to measure such characteristics in music generation remains an open question . Therefore , we draw inspiration from ( Yang & Lerch , 2018 ) to capture the style of a given performance based its the pitch- and rhythm-related features using 8 features : 1 . Note Density ( ND ) : The note density refers to the average number of notes per second in a performance : a higher note density often indicates a fast-moving piece , while a lower note density correlates with softer , slower pieces . This feature is a good indicator for rhythm . 2 . Pitch Range ( PR ) : The pitch range denotes the difference between the highest and lowest semitones ( MIDI pitches ) in a given phrase . 3 . Mean Pitch ( MP ) / Variation of Pitch ( VP ) : Similar in vein to the pitch range ( PR ) , the average and overall variation of pitch in a musical performance captures whether the piece is played in a higher or lower octave . 4 . Mean Velocity ( MV ) / Variation of Velocity ( VV ) : The velocity of each note indicates how hard a key is pressed in a musical performance , and serves as a heuristic for overall volume . 5 . Mean Duration ( MD ) / Variation of Duration ( VD ) : The duration describes for how long each note is pressed in a performance , representing articulation , dynamics , and phrasing .

Summary: This paper presents a technique for encoding the high level “style” of pieces of symbolic music. The music is represented as a variant of the MIDI format. The main strategy is to condition a Music Transformer architecture on this global “style embedding”.  Additionally, the Music Transformer model is also conditioned on a combination of both “style” and “melody” embeddings to try and generate music “similar” to the conditioning melody but in the style of the performance embedding.