Title: HAPPIER: HIERARCHICAL POLYPHONIC MUSIC GENERATIVE RNN

Abstract: Generating polyphonic music with coherent global structure is a major challenge for automatic composition algorithms. The primary difficulty arises due to the inefficiency of models to recognize underlying patterns beneath music notes across different levels of time scales and remain long-term consistency while composing. Hierarchical architectures can capture and represent learned patterns in different temporal scales and maintain consistency over long time spans, and this corresponds to the hierarchical structure in music. Motivated by this, focusing on leveraging the idea of hierarchical models and improve them to fit the sequence modeling problem, our paper proposes HAPPIER: a novel HierArchical Poly-Phonic musIc gEnerative RNN. In HAPPIER, A higher 'measure level' learns correlations across measures and patterns for chord progressions, and a lower 'note level' learns a conditional distribution over the notes to generate within a measure. The two hierarchies operate at different clock rates: the higher one operates on a longer timescale and updates every measure, while the lower one operates on a shorter timescale and updates every unit duration. The two levels communicate with each other, and thus the entire architecture is trained jointly end-to-end by back-propagation. HAPPIER, profited from the strength of the hierarchical structure, generates polyphonic music with long-term dependencies compared to the state-of-the-art methods.

1. INTRODUCTION:

Drawing inspiration from aesthetic intuition and expressing it as music with specific domain knowledge, composing is a marvelous artistic process for musicians. Due to this uniqueness of composing, although not a new idea, automatic composing remains to be a challenging task. Recent advances in machine learning, especially in sequence generative models, have greatly enabled novel insights into algorithmic music generation and music data analysis (Boulanger-Lewandowski et al., 2012; Gu et al., 2015; Chung et al., 2015; Chu et al., 2016; Engel et al., 2017; Hadjeres et al., 2017; Teng et al., 2017; Jaques et al., 2017; Yu & Varshney, 2017; Thickstun et al., 2017; Roberts et al., 2018) .

However, recognizing underlying patterns beneath music across different levels of temporal abstraction while training, and remaining long-term consistency while composing, is still a fundamental challenge for automatic composition algorithms. Thus, most approaches suffer from the lack of coherent global structure of music generated: it lacks consistent theme or structure, and appears to be random and wandering (Oord et al., 2016) (p. 8) .

Designing hierarchical architectures to operate on different spatial and temporal resolution scales to ameliorate these problems in machine learning algorithms is not a new idea, and has achieved great success in various fields, including but not limited in the field of computer vision (Lazebnik et al., 2006) , reinforcement learning (Dayan & Hinton, 1993; Kulkarni et al., 2016) , and in the field of sequence modeling (Hihi & Bengio, 1995; Koutnik et al., 2014; Serban et al., 2016; Mehri et al., 2017) .

We note that these kinds of hierarchical architectures correspond with the hierarchical nature of music: the composing of movements and phrases, the progression of chords, and the organization of notes within a measure lie on different temporal resolution levels, from the lowest to the highest. Although sharing some common traits, the patterns of them differ from each other significantly. We present an analogy in writing articles here as an illustration: when writing an article, a mature writer begins from organizing key ideas of each paragraph given the central idea of the whole article, then he arranges the progression of sentences within a paragraph conditioning on the main idea of the paragraph, and finally writes down words in each sentence given the idea of the sentence with respect to grammar rules. Although the whole article is a unified entity, the patterns and rules within each level, however, vary from each other. Thus, an efficient learning algorithm should be aware of this difference and is supposed to utilize it.

Motivated by this intuition, we propose HAPPIER: a novel HierArchical PolyPhonic musIc gEnerative RNN model to generate music sequentially. For simplicity, we model music as two hierarchies: the higher level measure hierarchy and the lower level note hierarchy. We also assume that every measure in music is specified with a corresponding chord while composing. Under these assumptions, HAPPIER contains a higher-level LSTM (Hochreiter & Schmidhuber, 1997) , a variant of RNN, learning correlations across measures and patterns for chord progressions, and a lower-level LSTM learning a conditional distribution over notes. The two hierarchies operate at different clock rates: the higher one updates every measure, while the lower one updates every unit duration. The higher level LSTM gives guidance over the lower level by projecting conditioning vectors to the lower one, meanwhile the lower level LSTM summarizes its cell states to the higher level LSTM once a measure in order to keep the latter one informed. The entire architecture is trained jointly end-to-end by back-propagation. HAPPIER, gained strength from hierarchical architectures, generates polyphonic music which maintains long-term dependencies, and performs better in listening tests compared to the state-of-the-art methods. Music samples generated by HAPPIER are provided in Appendix C.

2. RELATED WORKS:

Designing automatic music generation algorithms always attracts great attention from researchers, even dating back to the 80s last century. Attempts include to compose with handcrafted constraints (Ebcioglu, 1988) and to design machine learning algorithms, neural networks for instance (Todd, 1989) . However, approaches merely with handcrafted rules generally have unsatisfactory performance because of the lack of variety in the music they generate. Thus, we adopt the learning approach in this work. Analyzing music data also gains great attention in the machine learning community especially recently (Yu & Varshney, 2017; Thickstun et al., 2017; Roberts et al., 2018) . These works contribute much either in the perspective of datasets or learning hidden representations for music.

There are two approaches to generate music with respect to the type of data representation: one approach generating raw audio (Oord et al., 2016; Engel et al., 2017) , and the other composing music notes, e.g. in the form of MIDI or piano-roll (Boulanger-Lewandowski et al., 2012; Hadjeres et al., 2017; Jaques et al., 2017) . Our work belongs to the latter one. In this approach, most works consider monophonic composing (Jaques et al., 2017) , but advances have also been made recently in the more challenging polyphonic composing task (Chu et al., 2016) . In the light of data representation, our work is similar to the work of Yang et al. (2017) , both including simplified polyphonic MIDI representations of a chord track and a melody track.

Recent advances in deep learning have enabled great progress in automatic music generation. Different network architectures and training algorithms have been designed, including Convolutional Neural Networks (Oord et al., 2016; Engel et al., 2017; Yang et al., 2017) , Recurrent Neural Networks (RNN) (Boulanger-Lewandowski et al., 2012; Chu et al., 2016; Hadjeres et al., 2017; Jaques et al., 2017; Liang et al., 2017; Lim et al., 2017; Mehri et al., 2017; Teng et al., 2017; Ycart & Benetos, 2017) , and Generative Adversarial Nets (Goodfellow et al., 2014) (Yang et al., 2017) . However, as most of these works have pointed out themselves, generating music with coherent global structure still remains to be a major challenge for automatic composition algorithms. This difficulty may be primarily attributed to the inefficiency of models to learn long-term consistency in music, which is actually one of the fundamental challenges for most of sequence modeling tasks. Some of these works have made attempts to tackle this problem. Dating back to the 90s last century, the LSTM (Hochreiter & Schmidhuber, 1997) architecture was proposed to encourage long-term memory in recurrent networks by presenting shortcut connections with gating functions. However, the melodies it generates still remain somewhat random (Jaques et al., 2017) (p. 1). Oord et al. (2016) designed dilated causal convolution architectures to enlarge the receptive fields of neurons in CNN to learn long-range consistencies. However, the generated samples still vary second-tosecond (p. 8). Jaques et al. (2017) regarded the generating process as a sequential decision making process, and adopted a reinforcement learning approach, combining data prior with domain specific knowledge as reinforcement learning rewards to enforce coherent global structures. However, this approach requires handcrafted knowledge of musical theories into the system, which may involve heavy efforts. Hadjeres et al. (2017) gained strength from pseudo-Gibbs sampling to iteratively tune generated chorales. However, since iterative approaches are much more slower than sequential approaches, this method has a rather low generating efficiency. This paper adopt a hierarchical approach to ameliorate the problem. Our work is related to Sam-pleRNN (Mehri et al., 2017) , although their work does not consider automatic music composition problem. Similar to their approach, we use LSTM and different parts of our model run at different clock rates. Unlike their work, the higher hierarchy of our model operates on measures and chords, instead of on frames (p. 2), which are mere concatenations of inputs of the lower hierarchy. We remark that with the higher hierarchy operating on measures and chords, our model can explicitly utilize the knowledge that there are significant differences between the patterns of chord progressions and that of note organizations. Besides, if the number of hierarchies increases, the approach of simply concatenating the inputs of the lower hierarchies as the inputs of the higher ones may not be tractable, instead, some kind of dimension reduction is expected. In this way, we can regard our input of chords as a summarizing of the notes within each measure. Our approach also differs from SampleRNN in that summarizing paths from the lower hierarchy to the higher one is included. The benefits of these paths will be discussed and measured in Section 3 and Section 4. Chu et al. (2016) also design hierarchical LSTM architectures for composing. However, they first generate melodies, and then accompany the melody with chords and percussion. Instead, we recommend inversely, which is delighted by the remark of Teng et al. (2017) (p. 2) that 'This mimics how classical western Roman-numeral harmony is taught to beginners: only after one has the underlying chord sequence, can one explain the melody in terms of chord tones, passing tones, appoggiaturas, and so on.' Besides, although hierarchical, their approach does not have different parts of the model operating on various temporal resolution scales, and thus still cannot efficiently model the structures of music at very different scales, which finally leads to the lack of hierarchical temporal structure in the music they generate (Teng et al., 2017) (p. 1). While although Teng et al. (2017) first generate chords, and then generate melody based on chords, their model is not end-to-end, and the subparts of their model also do not operate on different clock rates.

3. METHOD:

In this paper, we propose a hierarchical LSTM (Hochreiter & Schmidhuber, 1997) model HAPPIER for polyphonic music generation. For simplicity, we show a two-level hierarchical LSTM as an example of the model. The model can also be easily generalized to multiple hierarchies in the same way as the two-level model is constructed. Here, we represent polyphonic music as a combination of a chord track and a melody track, where every measure in the music is specified with a corresponding chord. An example of the representation is shown in Figure 1 . In previous works focusing on monophonic music generation e.g. (Jaques et al., 2017) , composition is generally modeled as a sequential process of generating notes. The process is modeled by a conditional distribution over the next note n i to generate given all the previous notes generated so far, a deep approximator f parametrized by θ, e.g. an LSTM, is used to approximate this distribution:

Figure 2 : Illustration of the HAPPIER architecture. The loop of the recurrent net is unrolled over time for visualization, and the figure contains 3 measure level loops. A higher-level LSTM learns correlations between measures, and a lower-level LSTM learns a conditional distribution over the notes to generate in each measure given conditioning vectors from the measure level LSTM as guidance. The lower level LSTM summarizes its cell states to the higher one once a measure in order to keep the latter one informed. Different hierarchies operate on different timescales and different temporal resolution levels. The entire architecture is trained jointly end-to-end by back-propagation.

p(n i |n 1 , n 2 , .., n i-1 ) ≈ f (n 1 , n 2 , .., n i ; θ) (1)

However, music composition is challenging to these approaches because music contains structures at very different scales, and thus melody generated by approaches with mere LSTM tends to be wandering and random, and lack of coherent global structures (Jaques et al., 2017) (p. 1).

HAPPIER ameliorates the problem by designing end-to-end hierarchical architectures, including a measure level LSTM and a note level LSTM. The two hierarchies operate at different clock rates: the higher one updates every measure, while the lower one updates every unit duration.

Our formulation of the problem is similar to Equation 1, except that we also adopt a hierarchical representation of music. We denote the chord of the i-th measure c i , the j-th note in the i-th measure n ij , the number of notes in the i-th measure N i , and the total number of measures in a piece of music N . The process is defined by jointly modeling Equation 2 via hierarchical LSTM:

EQUATION

Figure 2 gives an overview of the architecture. Details of the architecture will be presented in Section 3.1 and Section 3.2.

In correspondence with this hierarchical architecture, we use the following way to represent music data. The way is inspired by the work of Hadjeres et al. (2017) , which aligns different tracks well for polyphonic music data representation. In this way, notes are represented by their MIDI pitches and time ticks denoting their beginning and holding; while chords are represented by their pitch encodings and time ticks. Figure 3 below explains this representation method.

One-hot vectors of these representations are used for network inputs. There are 129 types of notes and 79 types of chords in total in the Nottingham dataset (Boulanger-Lewandowski et al., 2012) . Thus, the prediction problem can be formulated into combinations of classification problems. For simplicity, we still denote notes as n and chords as c in the following sections, rather than separating their pitch and time tick representations. Readers may refer to Appendix B for details.

To train the classifier, the sum of cross entropy losses of chords and notes between the predicted distribution and the actual target distribution is minimized, as shown below: While during the generating process, we sample from the predicted distribution to get the next token given all the tokens generated so far. The process is repeated iteratively to generate complete music.

EQUATION

3.1. NOTE LEVEL LSTM:

The note level LSTM (Note Cells in Figure 2 ) learns patterns for note generation in the melody track. It operates at a shorter timescale and a higher temporal resolution level. For each unit duration t ij , its hidden states H ij and cell states C ij are updated, given the previous note n i(j-1) and previous state vectors H i(j-1) and C i(j-1) , together with a conditioning vector cond i from the measure level LSTM as a guidance for maintaining long-term consistency:

H ij , C ij = LST M([H i(j-1) , C i(j-1) , n i(j-1) , cond i ]), j = 1 H ij , C ij = LST M([H (i-1)(Ni-1) , C (i-1)(Ni-1) , n (i-1)(Ni-1) , cond i ]), j = 1 (4) A conditional distribution of notes D(n ij ) is predicted based upon H ij : D(n ij ) = Sof tmax(Linear(H ij )) (5)

At the end of each measure i, the note level LSTM projects a summarizing vector sum i to the measure level LSTM based upon its current cell state C iNi :

EQUATION

3.2. MEASURE LEVEL LSTM:

The measure level LSTM (Measure Cells in Figure 2 ) learns long-term correlations between measures and also learns patterns of chord progressions. It operates at a longer timescale and a lower temporal resolution level. For each measure t i , its hidden states H i and cell states C i are updated, given the previous chord c i and its previous state vectors H i-1 and C i-1 , together with a summarizing vector sum i-1 from the note level LSTM to keep it informed:

EQUATION

A conditioning vector cond i for the note level LSTM is generated from its current hidden state H i , which is identical within a measure i:

EQUATION

3.3. HAPPIER: INTEGRATING HIERARCHIES:

Intuitively, the two hierarchies can operate on different levels, and can thus make it possible to recognize underlying patterns beneath music across different levels of temporal abstraction. Besides, since the measure level LSTM does not update frequently given its clock rate, it retains long-term correlations, and can thus generate chords with long-term dependencies and guide the note level LSTM to remain consistency over a long range of time steps by projecting conditioning vectors to it.

The two hierarchies interact mutually via the conditioning paths and the summarizing paths. This is also one of the major differences between HAPPIER and SampleRNN (Mehri et al., 2017) .

The measure level LSTM guides the note level LSTM to remain long-term consistency by projecting conditioning vectors to it, meanwhile the summarizing paths keep the measure level LSTM informed of events happened downward in the note level LSTM.

The summarizing paths and the conditioning paths as an entity can also be seen as shortcut connections for the note level LSTM. It is known that deep architectures with long forwarding paths involving too many multiplication operations generally suffer from the risk of gradient vanishing or explosion during back-propagation (Pascanu et al., 2012) . While shortcut connections can encourage proper back-propagation supervision signals by skipping these multiplication operations, which is the main reason for the outperforming of LSTM (Hochreiter & Schmidhuber, 1997) over conventional RNNs and ResNet (He et al., 2016) over non-residual deep CNNs.

In an LSTM architecture, the gates decide whether to let all the information pass through, when a gated shortcut is 'closed', the functions in the network represent non-short-cut functions, while when a gate is 'open', all the information just passes through, and gradients can thus be back-propagated constantly to keep correlations exist between neighboring tokens as well as between ones apart distantly.

However, the decision of whether to 'open' these gates is learned in an LSTM architecture, and tokens hundreds of time steps away generally lose their correlations because of too many times of 'closing' of these gates. On the contrary, HAPPIER always supports short-cut connections. Consider two note cells hundreds of unit durations away, despite their distance in the note level, it takes only several measures to correlate the two cells in the measure level LSTM: information first goes upward via the summarizing path, then after several time-steps in the measure level LSTM, which has slower clock rates, it can get downward back to the note level LSTM via the conditioning path. Thus, the cooperation of the summarizing paths and the conditioning paths will encourage HAPPIER to learn coherent global structures in music.

Note that SampleRNN (Mehri et al., 2017) does not support these kinds of skip connections, because it does not support information from the lower level LSTM to get upward. Also, without summarizing paths we proposed, the measure level LSTM is modeling p(c i |c 1 , c 2 , .., c i-1 ) instead of Equation 2, which means that chords are generated regardless of previous notes generated.

4. EXPERIMENTS:

Quantitative and qualitative results are shown in this section to measure HAPPIER's sequence modeling performance. We process around 1000 polyphonic tunes into our data representation format shown in Figure 3 from the publicly available Nottingham dataset (Boulanger-Lewandowski et al., 2012) for training and validation. Preprocessing and learning details are presented in Appendix B.

4.1. QUANTITATIVE ANALYSIS FOR PREDICTION:

We first evaluate HAPPIER's sequence modeling capability in a prediction setting. We measure HAPPIER's prediction Negative Log Likelihood (NLL) of the next token given all previous tokens. This does not sufficiently reflect whether HAPPIER can learn long-term consistency over music, but still serves as a quantitative comparison experiment to show how each part of it works.

In this setting, we construct three models for comparison experiments: an LSTM Baseline (only the note level LSTM), a HAPPIER model with only conditioning path, and another HAPPIER model with both conditioning and summarizing paths. The three models share exactly the same set of architecture details and hyper-parameters. The prediction NLL of the pitch and tick (Figure 3 ) output by the note level LSTM of the three models are shown in Table 1 . Chord NLL is not compared, for the baseline LSTM without hierarchy cannot handle the hierarchical data structure of both a chord track and a melody track. The result indicates that HAPPIER performs better for melody track generation than the LSTM Baseline in the prediction setting, and that the improvement can be mainly attributed to the prescence of the conditioning path. This Subsection gives qualitative analysis of the generated samples from HAPPIER to show that it generates music retaining long-term consistency. Generated samples are presented in Appendix C, Figure 4 serves as an example of them. In Figure 4 , the generated tune remains in D major (e.g. given the #C and #F) globally. The progression of chords also follows D major. The melody track returns to the tonic note of D every a while and the chord track returns to the tonic chord correspondingly. Generated samples from HAPPIER retain long-term consistency in this way, and we also suggest our readers to listen to the generated samples themselves with comparison to the state-of-the-art methods to judge this1 .

4.3. LISTENING TESTS:

We also evaluate the performance of HAPPIER by human listening tests. We compare generated samples by HAPPIER with those from RNN-RBM (Boulanger-Lewandowski et al., 2012) , Deep-Bach (Hadjeres et al., 2017) and SequenceTutor (Jaques et al., 2017) . These three works all publish their generated samples online (Appendix A). These publicly accessible samples are used for this experiment. RNN-RBM is trained on similar datasets as ours, while DeepBach and Sequence Tutor are trained on different datasets. Sequence Tutor is trained with more than 30000 MIDI files, which are not made publicly accessible, and DeepBach is trained with 352 Bach Chorales.

We conduct AB preference tests among the generated samples of HAPPIER and those of the 3 models mentioned above. We also conduct a test between our generated samples and samples from our training dataset. Each subject is shown a set of 4 pairs of samples, and each pair contains samples from two different models. Subjects then choose the preferred samples or choose not to prefer to any of them (no pref.) after listening. Our samples are rendered with GarageBand from the MIDI files the network generates.

Subjects are also asked to give information about their musical expertise. They could choose a category fits them best among Level 1, knowing little about music theories; Level 2, music lovers or instrument players; and Level 3, having received training in composition. For this experiment, 179 subjects take the test, among whom 103 in Level 1, 64 in Level 2, and 12 in Level 3. Experiment results are shown in Figure 5 . Results show that HAPPIER is more auditorily pleasing compared with the state-of-the-art approaches in machine learning community, and the generated samples from HAPPIER can be hardly distinguished from samples from the Nottingham dataset. 

5. CONCLUSION:

We have proposed a novel end-to-end hierarchical polyphonic music generative RNN: HAPPIER, which gains strength from the correspondence of its hierarchical structure and the hierarchical nature of music. The model learns long-term correlations and performs better in listening tests compared to the state-of-the-art methods. The work mainly focuses on the automatic composition problem, but our contribution is not limited there. We believe the HAPPIER approach of designing hierarchical deep recurrent models with sub-parts of it operating on different clock rates and on different temporal resolution levels will be promising for a number of sequence modeling applications, where conventional approaches generally suffer from their inefficiency of learning long-range correlations.

:

Links to the generated samples of the state-of-the-art methods are presented inAppendix A.

Figures and Tables:

Figure 1: An example of the described representation of polyphonic music. The piece of music is extracted from the Nottingham Dataset (Boulanger-Lewandowski et al., 2012).

Figure 3: Illustration of the data representation. Boxes in the same color indicate the tokens and their corresponding encodings in the data representation. The piece of music is extracted from the Nottingham Dataset (Boulanger-Lewandowski et al., 2012).

Figure 4: One extracted generated sample by HAPPIER. The full sample and some more samples are presented in Appendix C.

Figure 5: Experiment results for the listening test. 179 subjects are involved.

Prediction NLL (Validation)

Table content: <table><tr><td>Model</td><td colspan="2">Note Pitch NLL Note Tick NLL</td></tr><tr><td>LSTM Baseline</td><td>0.854</td><td>0.216</td></tr><tr><td>HAPPIER: Only Conditioning</td><td>0.830</td><td>0.212</td></tr><tr><td colspan="2">HAPPIER: Conditioning + Summarizing 0.843</td><td>0.214</td></tr><tr><td>4.2 QUALITATIVE ANALYSIS FOR GENERATION</td><td/><td/></tr></table>

Peer review (iclr2019):
 PRO*s: +good problem: generating polyphonic music with long-term structure +reasonable approach: modification of SampleRNN: makes sense CON*s: -doesn*t work. My fundamental critique of this paper is that, while the authors claim that their system " generates polyphonic music which maintains long-term dependencies", in fact what it generates it is not really polyphonic, nor-- more importantly-- does it demonstrate the kind of long-term structure present in the training set. 1) Polyphony: The model predicts a combination of monophonic melody plus chords (i.e. chord names such as "A+", "C7" etc). This is different from polyphony, in which the model would predict the actual voicing used for those chords. However, this could be seen as an error in terminology; if the authors claimed that they were predicting a monophonic musical voice plus chords, and they did that well, that would be absolutely fine. Generating a coherent melodic line that continues, along with the chords underneath it, would be a great achievement. However, that is not what happens here. For examples, in the provided examples, e.g. Measure 19 of Fig 4, Measures 1,3,4,5, ... of Fig 6, contain stylistically unusual combinations of chords and melodic lines. By "stylistically unusual", I simply mean that those examples are not consistent with the Nottingham dataset. Furthermore, in my subjective opinion, the examples that I listed above also just don*t really musically work. There is no question that in the right context, any of those particular combinations of chords and melody notes *could* be made to work: for example, the first measure of Fig 6 would be perfectly fine as the beginning of a different song. (E.g. it could be taken as a slight reharmonization of the opening of "lullaby of birdland", but that would require a coherent continuation. ) 2) Long-term structure: It seems to me that one of the key things that this paper sets out to do is to get strong long-term dependencies. The motivation for the SampleRNN-inspired approach is to have generation at multiple time scales, for example. However, there is no evidence in the presented examples of long-term structure. Consider Fig 6, for example. Where is the long-term structure? A D major chord is frequently repeated with occasional A7. That is reasonable but it does not necessarily demonstrate long-term structure, anymore than learning that "q" is often followed by "u" demonstrates long-term structure. There is no melodic motif, there is no sense of 4-bar phrasing (or any other recurring such pattern that I can tell). In fact, all of the samples shown (Fig 4, 6, 7, 8) all end up with the chord D major played most of the time, after what appears to be a bit more variation in the first few chords. The results of the listening test are strange to me (beyond some of the apples-to-oranges comparisons). I cannot comment on those without hearing the pairs of examples that were actually played. How were those pairs selected? At the moment, it does not seem worthwhile for this review to get into details about exactly how the system works, in light of the problematic output. If there is reason for me to do so, I would gladly oblige. The authors do make a variety of choices that appear to be fairly sensible. I would very much look forward to seeing a revised version of the system in future that produces the kind of output that the system is intended to produce (and described as producing).