Music Transformer: Generating Music with Long-Term Structure

1 INTRODUCTION . A musical piece often consists of recurring elements at various levels , from motifs to phrases to sections such as verse-chorus . To generate a coherent piece , a model needs to reference elements that came before , sometimes in the distant past , and then repeat , vary , and further develop them to create contrast and surprise . Intuitively , self-attention ( Parikh et al. , 2016 ) could be a good match for this task . Self-attention over its own previous outputs allows an autoregressive model to access any part of the previously generated output at every step of generation . By contrast , recurrent neural networks have to learn to proactively store elements to be referenced in a fixed size state or memory , making training potentially much more difficult . We believe that repeating self-attention in multiple , successive layers of a Transformer decoder ( Vaswani et al. , 2017 ) can help capture the multiple levels at which self-referential phenomena exist in music . In its original formulation , the Transformer relies on absolute position representations , using either positional sinusoids or learned position embeddings that are added to the per-position input representations . Recurrent and convolutional neural networks instead model position in relative terms : RNNs through their recurrence over the positions in their input , and CNNs by applying kernels that effectively choose which parameters to apply based on the relative position of the covered input representations . ∗Google AI Resident . Correspondence to : Cheng-Zhi Anna Huang < annahuang @ google.com > 1Samples are available for listening at https : //goo.gl/magenta/music-transformer-examples . Later blog post at https : //magenta.tensorflow.org/music-transformer Music has multiple dimensions along which relative differences arguably matter more than their absolute values ; the two most prominent are timing and pitch . To capture such pairwise relations between representations , Shaw et al . ( 2018 ) introduce a relation-aware version of self-attention which they use successfully to modulate self-attention by the distance between two positions . We extend this approach to capture relative timing and optionally also pitch , which yields improvement in both sample quality and perplexity for the JSB Chorales dataset . As opposed to the original Transformer , samples from a Transformer with our relative attention mechanism maintain the regular timing grid present in this dataset . The model furthermore captures global timing , giving rise to regular phrases . The original formulation of relative attention ( Shaw et al. , 2018 ) requires O ( L2D ) memory where L is the sequence length and D is the dimension of the model ’ s hidden state . This is prohibitive for long sequences such as those found in the Maestro dataset of human-performed virtuosic , classical piano music ( Hawthorne et al. , 2019 ) . In Section 3.4 , we show how to reduce the memory requirements to O ( LD ) , making it practical to apply relative attention to long sequences . The Maestro dataset consists of MIDI recorded from performances of competition participants , bearing expressive dynamics and timing on a less than 10-millisecond granularity . Discretizing time in a fixed grid on such a resolution would yield unnecessarily long sequences as not all events change on the same timescale . We hence adopt a sparse , MIDI-like , event-based representation from ( Oore et al. , 2018 ) , allowing a minute of music with a 10-millisecond resolution to be represented at lengths around 2K . This is in contrast to a 6K to 18K length that would be needed on a serialized multi-attribute fixed-grid representation . As position in sequence no longer corresponds to time , a priori it is not obvious that relative attention should work as well with such a representation . However , we will show in Section 4.2 that it does improve perplexity and sample quality over strong baselines . We speculate that idiomatic piano gestures such as scales , arpeggios and other motifs all exhibit a certain grammar and recur periodically , hence knowing their relative positional distances makes it easier to model this regularity . This inductive bias towards learning relational information , as opposed to patterns based on absolute position , suggests that the Transformer with relative attention could generalize beyond the lengths it was trained on , which our experiments in Section 4.2.1 confirm . 1.1 CONTRIBUTIONS . Domain contributions We show the first successful use of Transformers in generating music that exhibits long-term structure . Before our work , LSTMs were used at timescales of 15s ( ~500 tokens ) of piano performances ( Oore et al. , 2018 ) . Our work demonstrates that Transformers not only achieve state-of-the-art perplexity on modeling these complex expressive piano performances , but can also generate them at the scale of minutes ( thousands of tokens ) with remarkable internal consistency . Our relative self-attention formulation is essential to the model ’ s quality . In listening tests ( see Section 4.2.3 ) , samples from models with relative self-attention were perceived as more coherent than the baseline Transformer model ( Vaswani et al. , 2017 ) . Relative attention not only enables Transformers to generate continuations that elaborate on a given motif , but also to generalize and generate in consistent fashion beyond the length it was trained on ( see Section 4.2.1 ) . In a seq2seq setup , Transformers can generate accompaniments conditioned on melodies , enabling users to interact with the model . Algorithmic contributions The space complexity of the relative self-attention mechanism in its original formulation ( Shaw et al. , 2018 ) made it infeasible to train on sequences of sufficient length to capture long-range structure in longer musical compositions . To address this , we present a crucial algorithmic improvement to the relative self-attention mechanism , dramatically reducing its memory requirements from O ( L2D ) to O ( LD ) . For example , the memory consumption per layer is reduced from 8.5 GB to 4.2 MB ( per head from 1.1 GB to 0.52 MB ) for a sequence of length L = 2048 and hidden-state size D = 512 ( per head Dh = DH = 64 , where number of heads is H = 8 ) ( see Table 1 ) , allowing us to use GPUs to train the relative self-attention Transformer on long sequences . 2 RELATED WORK . Sequence models have been the canonical choice for modeling music , from Hidden Markov Models to RNNs and Long Short Term Memory networks ( e.g. , Eck & Schmidhuber , 2002 ; Liang , 2016 ; Oore et al. , 2018 ) , to bidirectional LSTMs ( e.g. , Hadjeres et al. , 2017 ) . Successful application of sequential models to polyphonic music often requires serializing the musical score or performance into a single sequence , for example by interleaving different instruments or voices . Alternatively , a 2D pianoroll-like representation ( see A.1 for more details ) can be decomposed into a sequence of multi-hot pitch vectors , and their joint probability distributions can be captured using Restricted Boltzmann Machines ( Smolensky , 1986 ; Hinton et al. , 2006 ) or Neural Autoregressive Distribution Estimators ( NADE ; Larochelle & Murray , 2011 ) . Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks ( GAN ; Goodfellow et al. , 2014 ) . ( e.g. , Dong et al. , 2018 ) or as orderless NADEs ( Uria et al. , 2014 ; 2016 ) ( e.g. , Huang et al. , 2017 ) . Lattner et al . ( 2018 ) use self-similarity in style-transfer fashion , where the self-similarity structure of a piece serves as a template objective for gradient descent to impose similar repetition structure on an input score . Self-attention can be seen as a generalization of self-similarity ; the former maps the input through different projections to queries and keys , and the latter uses the same projection for both . Dot-product self-attention is the mechanism at the core of the Transformer , and several recent works have focused on applying and improving it for image generation , speech , and summarization ( Parmar et al. , 2018 ; Povey et al. , 2018 ; Liu et al. , 2018 ) . A key challenge encountered by each of these efforts is scaling attention computationally to long sequences . This is because the time and space complexity of self-attention grows quadratically in the sequence length . For relative self-attention ( Shaw et al. , 2018 ) this is particularly problematic as the space complexity also grows linearly in the dimension , or depth , of the per-position representations . 3 MODEL . 3.1 DATA REPRESENTATION . We take a language-modeling approach to training generative models for symbolic music . Hence we represent music as a sequence of discrete tokens , with the vocabulary determined by the dataset . Datasets in different genres call for different ways of serializing polyphonic music into a single stream and also discretizing time . The JSB Chorale dataset consists of four-part scored choral music , which can be represented as a matrix where rows correspond to voices and columns to time discretized to sixteenth notes . The matrix ’ s entries are integers that denote which pitch is being played . This matrix can than be serialized in raster-scan fashion by first going down the rows and then moving right through the columns ( see A.1 for more details ) . Compared to JSB Chorale , the piano performance data in the Maestro dataset includes expressive timing information at much finer granularity and more voices . For the Maestro dataset we therefore use the performance encoding proposed by Oore et al . ( 2018 ) which consists of a vocabulary of 128 NOTE_ON events , 128 NOTE_OFFs , 100 TIME_SHIFTs allowing for expressive timing at 10ms and 32 VELOCITY bins for expressive dynamics ( see A.2 for more details ) . 3.2 BACKGROUND : SELF-ATTENTION IN TRANSFORMER . The Transformer decoder is a autoregressive generative model that uses primarily self-attention mechanisms , and learned or sinusoidal position information . Each layer consists of a self-attention sub-layer followed by a feedforward sub-layer . The attention layer first transforms a sequence of L D-dimensional vectors X = ( x1 , x2 , . . . , xL ) into queries Q = XWQ , keys K = XWK , and values V = XWV , where WQ , WK , and WV are each D ×D square matrices . Each L×D query , key , and value matrix is then split into H L×Dh parts or attention heads , indexed by h , and with dimension Dh = DH , which allow the model to focus on different parts of the history . The scaled dot-product attention computes a sequence of vector outputs for each head as Zh = Attention ( Qh , Kh , V h ) = Softmax ( QhKh > √ Dh ) V h. ( 1 ) The attention outputs for each head are concatenated and linearly transformed to get Z , a L by D dimensional matrix . A upper triangular mask ensures that queries can not attend to keys later in the sequence . For other details of the Transfomer model , such as residual connections and learning rates , the reader can refer Vaswani et al . ( 2017 ) . The feedforward ( FF ) sub-layer then takes the output Z from the previous attention sub-layer , and performs two layers of point-wise dense layers on the depth D dimension , as shown in Equation 2 . W1 , W2 , b1 , b2 are weights and biases of those two layers . FF ( Z ) = ReLU ( ZW1 + b1 ) W2 + b2 ( 2 )

Summary: This paper describes a method for improving the (sequence-length) scalability of the Transformer architecture, with applications to modeling long-range interactions in musical sequences. The proposed improvement is applied to both global and local relative attention formulations of self-attention, and consists of a clever re-use (and re-shaping) of intermediate calculations. The result shaves a factor of L (sequence length) from the (relative) memory consumption, facilitating efficient training of long sequences. The method is evaluated on MIDI(-like) data of Bach chorales and piano performances, and compares favorably to prior work in terms of perplexity and a human listener evaluation.