Music Transformer: Generating Music with Long-Term Structure

1 INTRODUCTION . A musical piece often consists of recurring elements at various levels , from motifs to phrases to sections such as verse-chorus . To generate a coherent piece , a model needs to reference elements that came before , sometimes in the distant past , and then repeat , vary , and further develop them to create contrast and surprise . Intuitively , self-attention ( Parikh et al. , 2016 ) could be a good match for this task . Self-attention over its own previous outputs allows an autoregressive model to access any part of the previously generated output at every step of generation . By contrast , recurrent neural networks have to learn to proactively store elements to be referenced in a fixed size state or memory , making training potentially much more difficult . We believe that repeating self-attention in multiple , successive layers of a Transformer decoder ( Vaswani et al. , 2017 ) can help capture the multiple levels at which self-referential phenomena exist in music . In its original formulation , the Transformer relies on absolute position representations , using either positional sinusoids or learned position embeddings that are added to the per-position input representations . Recurrent and convolutional neural networks instead model position in relative terms : RNNs through their recurrence over the positions in their input , and CNNs by applying kernels that effectively choose which parameters to apply based on the relative position of the covered input representations . ∗Google AI Resident . Correspondence to : Cheng-Zhi Anna Huang < annahuang @ google.com > 1Samples are available for listening at https : //goo.gl/magenta/music-transformer-examples . Later blog post at https : //magenta.tensorflow.org/music-transformer Music has multiple dimensions along which relative differences arguably matter more than their absolute values ; the two most prominent are timing and pitch . To capture such pairwise relations between representations , Shaw et al . ( 2018 ) introduce a relation-aware version of self-attention which they use successfully to modulate self-attention by the distance between two positions . We extend this approach to capture relative timing and optionally also pitch , which yields improvement in both sample quality and perplexity for the JSB Chorales dataset . As opposed to the original Transformer , samples from a Transformer with our relative attention mechanism maintain the regular timing grid present in this dataset . The model furthermore captures global timing , giving rise to regular phrases . The original formulation of relative attention ( Shaw et al. , 2018 ) requires O ( L2D ) memory where L is the sequence length and D is the dimension of the model ’ s hidden state . This is prohibitive for long sequences such as those found in the Maestro dataset of human-performed virtuosic , classical piano music ( Hawthorne et al. , 2019 ) . In Section 3.4 , we show how to reduce the memory requirements to O ( LD ) , making it practical to apply relative attention to long sequences . The Maestro dataset consists of MIDI recorded from performances of competition participants , bearing expressive dynamics and timing on a less than 10-millisecond granularity . Discretizing time in a fixed grid on such a resolution would yield unnecessarily long sequences as not all events change on the same timescale . We hence adopt a sparse , MIDI-like , event-based representation from ( Oore et al. , 2018 ) , allowing a minute of music with a 10-millisecond resolution to be represented at lengths around 2K . This is in contrast to a 6K to 18K length that would be needed on a serialized multi-attribute fixed-grid representation . As position in sequence no longer corresponds to time , a priori it is not obvious that relative attention should work as well with such a representation . However , we will show in Section 4.2 that it does improve perplexity and sample quality over strong baselines . We speculate that idiomatic piano gestures such as scales , arpeggios and other motifs all exhibit a certain grammar and recur periodically , hence knowing their relative positional distances makes it easier to model this regularity . This inductive bias towards learning relational information , as opposed to patterns based on absolute position , suggests that the Transformer with relative attention could generalize beyond the lengths it was trained on , which our experiments in Section 4.2.1 confirm . 1.1 CONTRIBUTIONS . Domain contributions We show the first successful use of Transformers in generating music that exhibits long-term structure . Before our work , LSTMs were used at timescales of 15s ( ~500 tokens ) of piano performances ( Oore et al. , 2018 ) . Our work demonstrates that Transformers not only achieve state-of-the-art perplexity on modeling these complex expressive piano performances , but can also generate them at the scale of minutes ( thousands of tokens ) with remarkable internal consistency . Our relative self-attention formulation is essential to the model ’ s quality . In listening tests ( see Section 4.2.3 ) , samples from models with relative self-attention were perceived as more coherent than the baseline Transformer model ( Vaswani et al. , 2017 ) . Relative attention not only enables Transformers to generate continuations that elaborate on a given motif , but also to generalize and generate in consistent fashion beyond the length it was trained on ( see Section 4.2.1 ) . In a seq2seq setup , Transformers can generate accompaniments conditioned on melodies , enabling users to interact with the model . Algorithmic contributions The space complexity of the relative self-attention mechanism in its original formulation ( Shaw et al. , 2018 ) made it infeasible to train on sequences of sufficient length to capture long-range structure in longer musical compositions . To address this , we present a crucial algorithmic improvement to the relative self-attention mechanism , dramatically reducing its memory requirements from O ( L2D ) to O ( LD ) . For example , the memory consumption per layer is reduced from 8.5 GB to 4.2 MB ( per head from 1.1 GB to 0.52 MB ) for a sequence of length L = 2048 and hidden-state size D = 512 ( per head Dh = DH = 64 , where number of heads is H = 8 ) ( see Table 1 ) , allowing us to use GPUs to train the relative self-attention Transformer on long sequences . 2 RELATED WORK . Sequence models have been the canonical choice for modeling music , from Hidden Markov Models to RNNs and Long Short Term Memory networks ( e.g. , Eck & Schmidhuber , 2002 ; Liang , 2016 ; Oore et al. , 2018 ) , to bidirectional LSTMs ( e.g. , Hadjeres et al. , 2017 ) . Successful application of sequential models to polyphonic music often requires serializing the musical score or performance into a single sequence , for example by interleaving different instruments or voices . Alternatively , a 2D pianoroll-like representation ( see A.1 for more details ) can be decomposed into a sequence of multi-hot pitch vectors , and their joint probability distributions can be captured using Restricted Boltzmann Machines ( Smolensky , 1986 ; Hinton et al. , 2006 ) or Neural Autoregressive Distribution Estimators ( NADE ; Larochelle & Murray , 2011 ) . Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks ( GAN ; Goodfellow et al. , 2014 ) . ( e.g. , Dong et al. , 2018 ) or as orderless NADEs ( Uria et al. , 2014 ; 2016 ) ( e.g. , Huang et al. , 2017 ) . Lattner et al . ( 2018 ) use self-similarity in style-transfer fashion , where the self-similarity structure of a piece serves as a template objective for gradient descent to impose similar repetition structure on an input score . Self-attention can be seen as a generalization of self-similarity ; the former maps the input through different projections to queries and keys , and the latter uses the same projection for both . Dot-product self-attention is the mechanism at the core of the Transformer , and several recent works have focused on applying and improving it for image generation , speech , and summarization ( Parmar et al. , 2018 ; Povey et al. , 2018 ; Liu et al. , 2018 ) . A key challenge encountered by each of these efforts is scaling attention computationally to long sequences . This is because the time and space complexity of self-attention grows quadratically in the sequence length . For relative self-attention ( Shaw et al. , 2018 ) this is particularly problematic as the space complexity also grows linearly in the dimension , or depth , of the per-position representations . 3 MODEL . 3.1 DATA REPRESENTATION . We take a language-modeling approach to training generative models for symbolic music . Hence we represent music as a sequence of discrete tokens , with the vocabulary determined by the dataset . Datasets in different genres call for different ways of serializing polyphonic music into a single stream and also discretizing time . The JSB Chorale dataset consists of four-part scored choral music , which can be represented as a matrix where rows correspond to voices and columns to time discretized to sixteenth notes . The matrix ’ s entries are integers that denote which pitch is being played . This matrix can than be serialized in raster-scan fashion by first going down the rows and then moving right through the columns ( see A.1 for more details ) . Compared to JSB Chorale , the piano performance data in the Maestro dataset includes expressive timing information at much finer granularity and more voices . For the Maestro dataset we therefore use the performance encoding proposed by Oore et al . ( 2018 ) which consists of a vocabulary of 128 NOTE_ON events , 128 NOTE_OFFs , 100 TIME_SHIFTs allowing for expressive timing at 10ms and 32 VELOCITY bins for expressive dynamics ( see A.2 for more details ) . 3.2 BACKGROUND : SELF-ATTENTION IN TRANSFORMER . The Transformer decoder is a autoregressive generative model that uses primarily self-attention mechanisms , and learned or sinusoidal position information . Each layer consists of a self-attention sub-layer followed by a feedforward sub-layer . The attention layer first transforms a sequence of L D-dimensional vectors X = ( x1 , x2 , . . . , xL ) into queries Q = XWQ , keys K = XWK , and values V = XWV , where WQ , WK , and WV are each D ×D square matrices . Each L×D query , key , and value matrix is then split into H L×Dh parts or attention heads , indexed by h , and with dimension Dh = DH , which allow the model to focus on different parts of the history . The scaled dot-product attention computes a sequence of vector outputs for each head as Zh = Attention ( Qh , Kh , V h ) = Softmax ( QhKh > √ Dh ) V h. ( 1 ) The attention outputs for each head are concatenated and linearly transformed to get Z , a L by D dimensional matrix . A upper triangular mask ensures that queries can not attend to keys later in the sequence . For other details of the Transfomer model , such as residual connections and learning rates , the reader can refer Vaswani et al . ( 2017 ) . The feedforward ( FF ) sub-layer then takes the output Z from the previous attention sub-layer , and performs two layers of point-wise dense layers on the depth D dimension , as shown in Equation 2 . W1 , W2 , b1 , b2 are weights and biases of those two layers . FF ( Z ) = ReLU ( ZW1 + b1 ) W2 + b2 ( 2 )

Summary: This paper presents an implementation trick to reduce the memory footprint of relative attention within a transformer network. Specifically, the paper points out redudant computation and storage in the traditional implementation and re-orders matrix operations and indexing schemes to optimize. As an appllication, the paper applies the new implementation to music modeling and generation. By reducing the memory footprint, the paper is able to train the transformer with relative attention on longer musical sequences and larger corpora. The experimental results are compelling -- the transformer with relative attention outperforms baselines in terms of perplexity on development data (though test performance is not reported) and by manual evaluation in a user study.