Encoding Musical Style with Transformer Autoencoders

1 INTRODUCTION . There has been significant progress in generative modeling , particularly with respect to creative applications such as art and music ( Oord et al. , 2016 ; Engel et al. , 2017b ; Ha & Eck , 2017 ; Huang et al. , 2019a ; Payne , 2019 ) . As the number of generative applications increase , it becomes increasingly important to consider how users can interact with such systems , particularly when the generative model functions as a tool in their creative process ( Engel et al. , 2017a ; Gillick et al. , 2019 ) To this end , we consider how one can learn high-level controls over the global structure of a generated sample . We focus on symbolic music generation , where Music Transformer ( Huang et al. , 2019b ) is the current state-of-the-art in generating high-quality samples that span over a minute in length . The challenge in controllable sequence generation is the fact that Transformers ( Vaswani et al. , 2017 ) and their variants excel as language models or in sequence-to-sequence tasks such as translation , but it is less clear as to how they can : ( 1 ) learn and ( 2 ) incorporate global conditioning information at inference time . This contrasts with traditional generative models for images such as the variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) or generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) which typically incorprate global conditioning as part of their training procedure ( Sohn et al. , 2015 ; Sønderby et al. , 2016 ; Isola et al. , 2017 ; Van den Oord et al. , 2016 ) . In this work , we introduce the Transformer autoencoder , where we aggregate encodings across time to obtain a holistic representation of the performance style . We show that this learned global representation can be incorporated with other forms of structural conditioning in two ways . First , we show that given a performance , our model can generate performances that are similar in style to the provided input . Then , we explore different methods to combine melody and performance representations to harmonize a melody in the style of the given performance . In both cases , we show that combining both global and fine-scale encodings of the musical performance allows us to gain better control of generation , separately manipulating both the style and melody of the resulting sample . Empirically , we evaluate our model on two datasets : the publicly-available MAESTRO ( Hawthorne et al. , 2019 ) dataset , and an internal dataset of piano performances transcribed from 10,000+ hours of audio ( Anonymous for review ) . We find that the Transformer autoencoder is able to generate not only performances that sound similar to the input , but also accompaniments of melodies that follow a given style , as shown through both quantitative and qualitative experiments as well as a user listening study . In particular , we demonstrate that our model is capable of adapting to a particular musical style even in the case where we have one single input performance . 2 PRELIMINARIES . 2.1 DATA REPRESENTATION FOR MUSIC GENERATION . The MAESTRO ( Hawthorne et al. , 2019 ) dataset consists of over 1,100 classical piano performances , where each piece is represented as a MIDI file . The internal performance dataset consists of over 10,000 hours of piano performances transcribed from audio ( Anonymous for review ) . In both cases , we represent music as a sequence of discrete tokens , effectively formulating the generation task as a language modeling problem . The performances are encoded using the vocabulary as described in ( Oore et al. , 2018 ) , which captures expressive dynamics and timing . This performance encoding vocabulary consists of 128 note on events , 128 note off events , 100 time shift events representing time shifts in 10ms increments from 10ms to 1s , and 32 quantized velocity bins representing the velocity at which the 128 note on events were played . 2.2 MUSIC TRANSFORMER . We build our Transformer autoencoder from Music Transformer , a state-of-the-art generative model that is capable of generating music with long-term coherence ( Huang et al. , 2019b ) . While the original Transformer uses a self-attention mechanism that operates over absolute positional encodings of each token in a given sequence ( Vaswani et al. , 2017 ) , Music Transformer replaces this with relative attention ( Shaw et al. , 2018 ) , which allows the model to keep better track of regularity based on event orderings and periodicity in the performance . Huang et al . ( 2019b ) propose a novel algorithm for implementing relative self-attention that is significantly more memory-efficient , enabling the model to generate musical sequences over a minute in length . For more details regarding the self-attention mechanism and Transformers , we refer the reader to ( Vaswani et al. , 2017 ; Parmar et al. , 2018 ) . 3 CONDITIONAL GENERATION WITH THE TRANSFORMER AUTOENCODER . 3.1 MODEL ARCHITECTURE . We leverage the standard encoder and decoder stacks of the Transformer as a foundation for our model , with minor modifications that we outline below . Transformer Encoder : For both the performance and melody encoder networks , we use the Transformer ’ s stack of 6 layers which are each comprised of a : ( 1 ) multi-head relative attention mechanism ; and a ( 2 ) position-wise fully-connected feed-forward network . The performance encoder takes as input the event-based performance encoding of an input performance , while the melody encoder learns an encoding of the melody which has been extracted from the input performance . Depending on the music generation task , which we elaborate upon in Section 3.2 , the encoder output ( s ) are fed into the Transformer decoder . Figure 1 describes the way in which the encoder and decoder networks are composed together . Transformer Decoder : The decoder shares the same structure as the encoder network , but with an additional multi-head attention layer over the encoder outputs . At each step of generation , the decoder takes in the output of the encoder , as well as each new token that was previously generated . The model is trained end-to-end with maximum likelihood . That is , for a given sequence x of length n , we maximize log pθ ( x ) = ∑n i=1 log pθ ( xi|x < i ) with respect to the model parameters θ . 3.2 CONDITIONING MECHANISM . Performance Conditioning and Bottleneck For this task , we aim to generate samples that sound “ similar ” to a conditioning input performance . We incorporate a bottleneck in the output of the Transformer encoder in order to prevent the model from simply memorizing the input ( Baldi , 2012 ) . Thus , as shown in Figure 1 , we mean-aggregate the performance embedding across the time dimension in order to learn a global representation of style . This mean-performance embedding is then fed into the autoregressive decoder , where the decoder attends to this global representation in order to predict the appropriate target . Although this bottleneck may be undesirable in sequence transduction tasks where the input and output sequences differ ( e.g . translation ) , we find that it works well in our setting where we require the generated samples to be similar in style to the input sequence . Melody & Performance Conditioning : Next , we synthesize any given melody in the style of a different performance . Although the setup shares similarities to that of the melody conditioning problem in ( Huang et al. , 2019b ) , we note that we also provide a conditioning performance signal , which makes the generation task more challenging . During training , we follow an internal procedure to extract melodies from performances in the training set , quantize the melody to a 100ms grid , and encode it as a sequence of tokens that uses a different vocabulary than the performance representation . We then use two distinct Transformer encoders ( each with the same architecture ) as in Section 3.1 to separately encode the melody and performance inputs . The melody and performance embeddings are combined to use as input to the decoder . We explore various ways of combining the intermediate representations : ( 1 ) sum , where we add the performance and melody embeddings together ; ( 2 ) concatenate , where we concatenate the two embeddings separated with a stop token ; and ( 3 ) tile , where we tile the performance embedding across every dimension of time in the melody encoding . In all three cases , we work with the meanaggregated representation of the input performance . We find that different approaches work better than others on some dataets , a point which we elaborate upon in Section 5 . Input Perturbation In order to encourage the encoded performance representations to generalize across various melodies , keys , and tempos , we draw inspiration from the denoising autoencoder ( Vincent et al. , 2008 ) as a means to regularize the model . For every target performance from which we extract the input melody , we provide the model with a perturbed version of the input performance as the conditioning signal . We allow this “ noisy ” performance to vary across two axes of variation : ( 1 ) pitch , where we artificially shift the overall pitch either down or up by 6 semitones ; and ( 2 ) time , where we stretch the timing of the performance by at most 5 % . In our experiments , we find that this augmentation procedure leads to samples that sound more pleasing ( Oore et al. , 2018 ) . We provide further details on the augmentation procedure in Appendix A . 4 SIMILARITY EVALUATION ON PERFORMANCE FEATURES . Although a variety of different metrics have been proposed to quantify both the quality ( Engel et al. , 2019 ) and similarity of musical performances relative to one another ( Yang & Lerch , 2018 ; Hung et al. , 2019 ) , the development of a proper metric to measure such characteristics in music generation remains an open question . Therefore , we draw inspiration from ( Yang & Lerch , 2018 ) to capture the style of a given performance based its the pitch- and rhythm-related features using 8 features : 1 . Note Density ( ND ) : The note density refers to the average number of notes per second in a performance : a higher note density often indicates a fast-moving piece , while a lower note density correlates with softer , slower pieces . This feature is a good indicator for rhythm . 2 . Pitch Range ( PR ) : The pitch range denotes the difference between the highest and lowest semitones ( MIDI pitches ) in a given phrase . 3 . Mean Pitch ( MP ) / Variation of Pitch ( VP ) : Similar in vein to the pitch range ( PR ) , the average and overall variation of pitch in a musical performance captures whether the piece is played in a higher or lower octave . 4 . Mean Velocity ( MV ) / Variation of Velocity ( VV ) : The velocity of each note indicates how hard a key is pressed in a musical performance , and serves as a heuristic for overall volume . 5 . Mean Duration ( MD ) / Variation of Duration ( VD ) : The duration describes for how long each note is pressed in a performance , representing articulation , dynamics , and phrasing .

Summary: In this paper, the author extends the standard music Transformer into a conditional version: two encoders are evolved, one for encoding the performance and the other is used for encoding the melody. The output representation has to be similar to the input. The authors conduct experiments on the MAESTRO dataset and an internal, 10,000+ hour dataset of piano performances to verify the proposed algorithm.