A Hierarchical Recurrent Neural Network for Symbolic Melody Generation

In recent years, neural networks have been used to generate symbolic melodies. However, the long-term structure in the melody has posed great difficulty for designing a good model. In this paper, we present a hierarchical recurrent neural network for melody generation, which consists of three Long-Short-Term-Memory (LSTM) subnetworks working in a coarse-to-fine manner along time. Specifically, the three subnetworks generate bar profiles, beat profiles and notes in turn, and the output of the high-level subnetworks are fed into the low-level subnetworks, serving as guidance for generating the finer time-scale melody components in low-level subnetworks. Two human behavior experiments demonstrate the advantage of this structure over the single-layer LSTM which attempts to learn all hidden structures in melodies. Compared with the state-of-the-art models MidiNet (Yang, Chou, and Yang 2017) and MusicVAE (Roberts et al. 2018), the hierarchical recurrent neural network produces better melodies evaluated by humans.  Automatic music generation using neural networks has attracted much attention. There are two classes of music generation approaches, symbolic music generation (Hadjeres, Pachet, and Nielsen 2017)(Waite et al. 2016)(Yang, Chou, and Yang 2017) and audio music generation (van den Oord et al. )(Mehri et al. 2016). In this study, we focus on symbolic melody generation, which requires learning from sheet music. Many music genres such as pop music consist of melody and harmony. Since usually beautiful harmonies can be ensured by using legitimate chord progressions which have been summarized by musicians, we only focus on melody generation, similar to some recent studies (Waite et al. 2016)(Yang, Chou, and Yang 2017)(Colombo, Seeholzer, and Gerstner 2017)(Roberts et al. 2018). This greatly simplifies the melody generation problem. Melody is a linear succession of musical notes along time. It has both short time scale such as notes and long time scale such as phrases and movements, which makes the melody generation a challenging task. Existing methods generate pitches and rhythm simultaneously (Waite et al. 2016) or sequentially (Chu, Urtasun, and Fidler 2016) using Recurrent Neural Networks (RNNs), but they usually work on the note scale without explicitly modeling the larger time-scale components such as rhythmic patterns. It is difficult for them to learn long-term dependency or structure in melody. Theoretically, an RNN can learn the temporal structure of any length in the input sequence, but in reality, as the sequence gets longer it is very hard to learn long-term structure. Different RNNs have different learning capability, e.g., LSTM (Hochreiter and Schmidhuber 1997) performs much better than the simple Elman network. But any model has a limit for the length of learnable structure, and this limit depends on the complexity of the sequence to be learned. To enhance the learning capability of an RNN, one approach is to invent a new structure. In this work we take another approach: increase the granularity of the input. Since each symbol in the sequence corresponds to longer segment than the original representation, the same model would learn longer temporal structure. To implement this idea, we propose a Hierarchical Recurrent Neural Network (HRNN) for learning melody. It consists of three LSTM-based sequence generators — Bar Layer, Beat Layer and Note Layer. The Bar Layer and Beat Layer are trained to generate bar profiles and beat profiles, which are designed to represent the high-level temporal features of melody. The Note Layer is trained to generate melody conditioned on the bar profile sequence and beat profile sequence output by the Bar Layer and Beat Layer. By learning on different time scales, the HRNN can grasp the general regular patterns of human composed melodies in different granularities, and generate melody with realistic long-term structures. This method follows the general idea of granular computing (Bargiela and Pedrycz 2012), in which different resolutions of knowledge or information is extracted and represented for problem solving. With the shorter profile sequences to guide the generation of note sequence, the difficulty of generating note sequence with wellorganized structure is alleviated. Related Work Melody Generation with Neural Networks There is a long history of generating melody with RNNs. A recurrent autopredictive connectionist network called CONCERT is used to compose music (Mozer 1994). With a set of composition rules as constraints to evaluate ar X iv :1 71 2. 05 27 4v 2 [ cs .S D ] 5 S ep 2 01 8 melodies, an evolving neural network is employed to create melodies (Chen and Miikkulainen 2001). As an important form of RNN, LSTM (Hochreiter and Schmidhuber 1997) is used to capture the global music structure and improve the quality of the generated music (Eck and Schmidhuber 2002). Boulanger-Lewandowski, Bengio, and Vincent explore complex polyphonic music generation with an RNNRBM model (Boulanger-Lewandowski, Bengio, and Vincent 2012). Lookback RNN and Attention RNN are proposed to tackle the problem of creating melody’s long-term structure (Waite et al. 2016). The Lookback RNN introduces a handcrafted lookback feature that makes the model repeat sequences easier while the Attention RNN leverages an attention mechanism to learn longer-term structures. Inspired by convolution, two variants of RNN are employed to attain transposition invariance (Johnson 2017). To model the relation between rhythm and melody flow, a melody is divided into pitch sequence and duration sequence and these two sequences are processed in parallel (Colombo et al. 2016). This approach is further extended in (Colombo, Seeholzer, and Gerstner 2017). A hierarchical VAE is employed to learn the distribution of melody pieces in (Roberts et al. 2018), the decoder of which is similar to our model. The major difference is that the higher layer of its decoder uses the automatically learned representation of bars, while our higher layers use predefined representation of bars and beats which makes the learning problem easier. Generative Adversarial Networks (GANs) have also been used to generate melodies. For example, RNN-based GAN (Mogren 2016) and CNN-based GAN (Yang, Chou, and Yang 2017) are employed to generate melodies, respectively. However, the generated melodies also lack realistic long-term structures. Some models are proposed to generate multi-track music. A 4-layer LSTM is employed to produce the key, press, chord and drum of pop music seperately (Chu, Urtasun, and Fidler 2016). With pseudo-Gibbs sampling, a model can generate highly convincing chorales in the style of Bach (Colombo, Seeholzer, and Gerstner 2017). Three GANs for symbolic-domain multi-track music generation were proposed (Dong et al. 2018). An end-to-end melody and arrangement generation framework XiaoIce Band was proposed to generate a melody track with accompany tracks with RNN (Zhu et al. 2018). Hierarchical and Multiple Time Scales Networks The idea of hierarchical or multiple time scales has been used in neural network design, especially in the area of natural language processing. The Multiple Timescale Recurrent Neural Network (MTRNN) realizes the self-organization of a functional hierarchy with two types of neurons “fast” unit and “slow” unit (Yamashita and Tani 2008). Then it is shown that the MTRNN can acquire the capabilities to recognize, generate, and correct sentences in a hierarchical way: characters grouped into words, and words into sentences (Hinoshita et al. 2011). An LSTM auto-encoder is trained to preserve and reconstruct paragraphs by hierarchically building embeddings of words, sentences and paragraphs (Li, Luong, and Jurafsky 2015). To process inputs at multiple time Time signature Bar Key signature