Venue: NAACL 2018

# A Melody-conditioned Lyrics Language Model

Kento Watanabe\({}^{1}\), Yuichiroh Matsubayashi\({}^{1}\),

**Satoru Fukayama\({}^{2}\)**, **Masataka Goto\({}^{2}\)**, **Kentaro Inui\({}^{1,3}\)**, **Tomoyasi Nakano\({}^{2}\)**

\({}^{1}\)Graduate School of Information Sciences, Tohoku University,

\({}^{2}\)National Institute of Advanced Industrial Science and Technology (AIST),

\({}^{3}\)RIKEN Center for Advanced Intelligence Project

{kento.w, y-matsu, inui}@ecei.tohoku.ac.jp,

{s.fukayama, m.goto, t.nakano}@aist.go.jp

###### Abstract

This paper presents a novel, data-driven language model that produces entire lyrics for a given input melody. Previously proposed models for lyrics generation suffer from the inability of capturing the relationship between lyrics and melody partly due to the unavailability of lyrics-melody aligned data. In this study, we first propose a new practical method for creating a large collection of lyrics-melody aligned data and then create a collection of 1,000 lyrics-melody pairs augmented with precise syllable-note alignments and word/sentence/paragraph boundaries. We then provide a quantitative analysis of the correlation between word/sentence/paragraph boundaries in lyrics and melodies. We then propose an RNN-based lyrics language model conditioned on a featurized melody. Experimental results show that the proposed model generates fluent lyrics while maintaining the compatibility between boundaries of lyrics and melody structures.

## 1 Introduction

Writing lyrics for a given melody is a challenging task. Unlike prose text, writing lyrics requires both knowledge and consideration of music-specific properties such as the structure of melody, rhythms, etc. Austin et al. (2010); Ueda (2010). A simple example is the correlation between word boundaries in lyrics and the rests in a melody. As shown in Figure 1, a single word spanning beyond a long melody rest can sound unnatural. When writing lyrics, a lyricist must consider such constraints in content and lexical selection, which can impose extra cognitive loads.

This consideration when writing lyrics has motivated a wide-range of studies for the task of computer-assisted lyrics writing Barbieri et al. (2012); Abe and Ito (2012); Potash et al. (2015); Watanabe et al. (2017). Such studies aim to model the language in lyrics and to design a computer system for assisting lyricists in writing. They propose to constrain their models to generate only lyrics that satisfy given conditions on syllable counts, rhyme positions, etc. However, such constraints are assumed to be manually provided by a human user, which requires the user to interpret a source melody and transform their interpretation to a set of constraints. To assist users with transforming a melody to constraints, a language model that automatically captures the relationship between lyrics and melody is required.

Some studies Oliveira et al. (2007); Oliveira (2015); Nichols et al. (2009) have quantitatively analyzed the correlations between melody and phonological aspects of lyrics (e.g., the relationship between a beat and a syllable stress). However, these studies do not address the relationship between melody and the _discourse structure_ of lyrics. Lyrics are not just a sequence of syllables but a meaningful sequence of words. Therefore, it is desirable that the sentence/paragraph boundaries are determined based on both melody rests and context words.

Considering such line/paragraph structure of lyrics, we present a novel language model that gen

Figure 1: Examples of awkward and natural lyrics. FUNC indicates a function word. The song is from the RWC Music Database (RWC-MDB-P-2001 No.20) Goto et al. (2002).



erates lyrics whose word, sentence, and paragraph boundaries are appropriate for a given melody, without manually transforming the melody to syllable constraints. This direction of research has received less attention because it requires a large dataset consisting of aligned pairs of melody and segment boundaries of lyrics which has yet to exist.

To address this issue, we leverage a publicly-available collection of digital music scores and create a dataset of digital music scores each of which specifics a melody score augmented with syllable information for each melody note. We collected 1,000 Japanese songs from an online forum where many amateur music composers upload their music scores. We then automatically aligned each music score with the raw text data of the corresponding lyrics in order to augment it with the word, sentence, and paragraph boundaries.

The availability of such aligned, parallel data opens a new area of research where one can conduct a broad range of data-oriented research for investigating and modeling correlations between melodies and discourse structure of lyrics. In this paper, with our melody-lyrics aligned songs, we investigate the phenomena that (i) words, sentences, and paragraphs rarely span beyond a long melody rest and (ii) the boundaries of larger components (i.e., paragraphs) tend to coincide more with longer rests. To the best of our knowledge, there is no previous work that provides any quantitative analysis of this phenomenon with this size of data (see Section 7).

Following this analysis, we build a novel, data-driven language model that generates fluent lyrics whose sentence and paragraph boundaries fit an input melody. We extend a Recurrent Neural Network Language Model (RNNLM) Mikolov et al. (2010) so that its output can be conditioned on a featurized melody. Both our quantitative and qualitative evaluations show that our model captures the consistency between melody and boundaries of lyrics while maintaining word fluency.

## 2 Melody-lytic alignment data

Our goal is to create a melody-conditioned language model that captures the correlations between melody patterns and discourse segments of lyrics. The data we need for this purpose is a collection of melody-lyrics pairs where the melody and lyrics are aligned at the level of not only notesyllable alignment but also discourse components (i.e., word/sentence/paragraph boundaries) of a lyric, as illustrated in the bottom of Figure 2. We create such a dataset by automatically combining two types of data available from online forum sites: digital music score data (the top of Figure 2) and raw lyrics data (the middle).

A digital music score specifies a melody score augmented with syllable information for each melody note (see the top of Figure 2). Score data augmented in this way is sufficient for analyzing the relationship between the phonological aspects of lyrics and melody, but it is insufficient for our goal since the structural information of the lyrics is not included. We thus augment score data further with boundaries of sentences, and paragraphs, where we assume that sentences and paragraphs of lyrics are approximately captured by _lines_ and _blocks_,1 respectively, of the lyrics in the raw text.

Footnote 1: Blocks are assumed to be segmented by empty lines.

The integration of music scores and raw lyrics is achieved by (1) applying a morphological analyzer2 to raw lyrics for word segmentation and Chinese character pronunciation prediction and (2) aligning music score with raw lyrics at the syllable level as illustrated in Figure 2. For this alignment, we employ the Needleman-Wunsch algorithm Needleman and Wunsch (1970). This alignment process is reasonably accurate because it fails in principle only when the morphological analysis fails in Chinese character pronunciation prediction, which occurs for only less than 1% of the words in the data set.

Footnote 2: To extract word boundaries and syllable information for Japanese lyrics, we apply McCab parser Kudo et al. (2004).

With this procedure, we obtained 54,181 Japanese raw lyrics and 1,000 digital musical

Figure 2: Melody-lyrics alignment using the Needleman Wunsch algorithm. BOL denotes a line boundary.

 

scores from online forum sites3; we thus created 1,000 melody-lyrics pairs. We refer to these 1,000 melody-lyrics pairs as a _melody- lyrics alignment_ data4 and refer to the remaining 53,181 lyrics without melody as a _raw lyrics_ data.

Footnote 3: For selecting the 1,000 songs, we chose only frequently downloaded or highly popular songs to ensure the quality of the resulting dataset.

Footnote 4: We publicly release all source URLs of the 1,000 songs ([https://github.com/KentoW/melody-lyrics](https://github.com/KentoW/melody-lyrics)).

We randomly split the 1,000 melody-lyrics alignments into two sets: 90% for analyzing/training and the remaining 10% for testing. From those, we use 20,000 of the most frequent words whose syllable counts are equal to or less than 10, and converted others to a special symbol \(\langle\text{unknown}\rangle\). All of the digital music score data we collected were distributed in the UST format, a common file format designed specifically for recently emerging computer vocal synthesizers. While we focus on Japanese music in this study, our method for data creation is general enough to be applied to other language formats such as MusicXML and ABC, because transferring such data formats to UST is straightforward.

## 3 Correlations between melody and lyric

In this section, we examine two phenomena related to boundaries of lyrics: (1) the positions of lyrics segment boundaries are biased to melody rest positions, and (2) the probability of boundary occurrence depends on the duration of a rest, i.e., a shorter rest tends to be a word boundary and a longer rest tends to be a block boundary, as shown in Figure 3. All analyses were performed on the training split of the melody-lyrics alignment data, which is described in Section 2.

For the first phenomenon, we first calculated the distribution of boundary appearances at the positions of melody notes and rests. Here, by the _boundary of a line_ (or block), we refer to the position of the beginning of the line (or block).5 In Figure 3, we say, for example, that the boundary of the first block beginning "_te-ra-shi te_" coincides with Rest#1. The result, shown at the top of Figure 4, indicates that line and block boundaries are strongly biased to rest positions and are far less likely to appear at note positions. Words, lines, and blocks rarely span beyond a long melody rest.

Footnote 5: The beginning of a line/block and the end of a line/block are equivalent since there is no melody between the end and beginning of a line/block.

The bottom of Figure 4 shows the detailed distributions of boundary occurrences for different durations of melody rests, where durations of \(480\) and \(1920\) correspond to a quarter rest and a whole rest, respectively. The results exhibit a clear, strong tendency that the boundaries of larger segments tend to coincide more with longer rests. To the best of our knowledge, this is the first study that has ever provided such strong empirical evidence for the phenomena related to the correlations between lyrics segments and melody rests. It is also important to note that the choice of segment boundaries looks like a probabilistic process (i.e., there is a long rest without a block boundary). This observation suggests the difficulty of describing the correlations of lyrics and melody in a rule-based fashion and motivates our probabilistic approach as we present in the next section.

## 4 Melody-conditioned language model

Our goal is to build a language model that generates fluent lyrics whose discourse segment fit a given melody in the sense that generated segment boundaries follow the distribution observed in Section 3. We propose to pursue this goal by conditioning a

Figure 4: Distribution of the number of boundaries in the melody-lyrics alignment data.

Figure 3: Example boundaries appearing immediately after a rest. BOB indicates a block boundary.

 

standard RNNLM with a featurized input melody. We call this model a _Melody-conditioned RNNLM_.

The network structure of the model is illustrated in Figure 5. Formally, we are given a melody \({\textbf{m}}=m_{1}\),...,\(m_{i}\),...,\(m_{I}\) that is a sequence of notes and rests, where \(m\) includes a pitch and a duration information. Our model generates lyrics \({\textbf{w}}=w_{1}\),...,\(w_{t}\),...,\(w_{T}\) that is a sequence of words and segment boundary symbols: \(\langle\text{BOL}\rangle\) and \(\langle\text{BOB}\rangle\), special symbols denoting a line and a block boundary, respectively. For each time step \(t\), the model outputs a single word or boundary symbol taking a pair of the previously generated word \(w_{t-1}\) and the musical feature vector \({\textbf{n}}_{t}\) for the current word position which includes context window-based features that we describe in the following section. In this model, we assume that the syllables of the generated words and the notes in the input melody have a one-to-one correspondence. Therefore, the position of the incoming note/rest for a word position \(t\) (referred to as a target note for \(t\)) is uniquely determined by the syllable counts of the previously generated words.6 The target note for \(t\) is denoted as \(m_{i(t)}\) by defining a function \(i(\cdot)\) which maps time step \(t\) to the index of the next note in \(t\).

Footnote 6: Note that our melody-lyrics alignment data used in training does not make this assumption, but we can still uniquely identify the positions of target notes based on the obtained melody-word alignment.

Here, the challenging issue with this model is training. Generally, language models require a large amount of text data to learn well. Moreover, this is also the case for learning correlation between rest positions and _syllable counts_. As shown in Figure 4, most words are supposed to not overlap a long rest. This means, for example, that when the incoming melody sequence for a next word position is _note, note, (long) rest, note, note_, as the sequence following to \(m_{i(t-1)}\) in Figure 5, it is desirable to select a word whose syllable count is two or less so that the generated word does not overlap the long rest. If there is sufficient data available, this tendency may be learned directly from the correlation between rests and words without explicitly considering the syllable count of a word. However, our melody-lyrics alignments for 1,000 songs are insufficient for this purpose.

We take two approaches to address this data sparsity problem. First, we propose two training strategies that increase the number of training examples using raw lyrics that can be obtained in greater quantities. Second, we construct a model that predicts the number of syllables in each word, as well as words themselves, to explicitly supervise the correspondence between rest positions and syllable counts.

In the following sections, we first describe the details of the proposed model and then present the training strategies used to obtain better models with our melody-lyrics alignment data.

### Model construction

The proposed model is based on a standard RNNLM Mikolov et al. (2010):

\[P({\textbf{w}})=\prod_{t=1}^{T}P(w_{t}|w_{0},...,w_{t-1}), \tag{1}\]

where context words are encoded using LSTM Hochreiter and Schmidhuber (1997) and the probabilities over words are calculated by a \(\operatorname*{softmax}\) function. \(w_{0}=\langle\mathrm{B}\rangle\) is a symbol denoting the beginning of lyrics. We extend this model such that each output is conditioned by the context melody vectors \({\textbf{n}}_{1},...,{\textbf{n}}_{t}\), as well as previous words:

\[P({\textbf{w}}|{\textbf{m}})=\prod_{t=1}^{T}P(w_{t}|w_{0},...,w_{ t-1},{\textbf{n}}_{1},...,{\textbf{n}}_{t}). \tag{2}\]

The model simultaneously predicts the syllable counts of words by sharing the parameters of LSTM with the above word prediction model in order to learn the correspondence between the melody segments and syllable counts:

\[P({\textbf{s}}|{\textbf{m}})=\prod_{t=1}^{T}P(s_{t}|w_{0},..., w_{t-1},{\textbf{n}}_{1},...,{\textbf{n}}_{t}), \tag{3}\]

where \({\textbf{s}}=s_{1},...,s_{T}\) is a sequence of syllable counts, which corresponds to **w**.

Figure 5: Melody-conditioned RNNLM.

 

For each time step \(t\), the model outputs a word distribution \({\bf y}_{u}^{t}\in\mathbb{R}^{V}\) and a distribution of syllable count \({\bf y}_{s}^{t}\in\mathbb{R}^{S}\) using a \(\mathrm{softmax}\) function:

\[{\bf y}_{w}^{t}=\mathrm{softmax}(\mathrm{BN}({\bf W}_{w}{\bf z}_{ t})), \tag{4}\] \[{\bf y}_{s}^{t}=\mathrm{softmax}(\mathrm{BN}({\bf W}_{s}{\bf z}_{ t})), \tag{5}\]

where \({\bf z}_{t}\) is the output of the LSTM for each time step. \(V\) is the vocabulary size and \(S\) is the syllable count threshold.7\({\bf W}_{w}\) and \({\bf W}_{s}\) are weight matrices. BN denotes batch normalization Ioffe and Szegedy (2015).

Footnote 7: The syllable counts of the \(\langle\text{BOL}\rangle\) and \(\langle\text{BOB}\rangle\) are zero.

The input to the LSTM in each time step \(t\) is a concatenation of the embedding vector of the previous word \({\bf v}(w_{t-1})\) and the context melody representation \({\bf x}_{n}^{t}\), which is a nonlinear transformation of the context melody vector \({\bf n}_{t}\):

\[{\bf x}^{t} = [{\bf v}(w_{t-1}),{\bf x}_{n}^{t}], \tag{6}\] \[{\bf x}_{n}^{t} = \mathrm{ReLU}({\bf W}_{n}{\bf n}_{t}+{\bf b}_{n}), \tag{7}\]

where \({\bf W}_{n}\) is a weight matrix and \({\bf b}_{n}\) is a bias.

To generate lyrics, the model searches for the word sequence with the greatest probability (Eq. 2) using beam search. The model stops generating lyrics when the syllable count of the lyrics reaches the number of notes in the input melody.

Note that our model is not specific to the language of lyrics. The model only requires the sequences of melody, words, and syllable counts and does not use any language-specific features.

### Context melody vector

In Section 3, we indicated that the positions of rests and their durations are important factors for modeling boundaries of lyrics. Thus, we collect a sequence of notes and rests around the current word position (i.e., time step \(t\)) and encode their information into context melody vector \({\bf n}_{t}\) (see the bottom of Figure 5).

```
1:for each syllable in the input-lyrics do
2:\(b\leftarrow\) get boundary type next to the syllable
3: sample note pitch \(p\sim P(p_{l}|p_{i-2},p_{i-1})\)
4: sample note duration \(d_{\text{note}}\sim P(d_{\text{note}}|b)\)
5: assign note with \((p,d_{\text{use}})\) to be the syllable
6: sample binary variable \(r\sim P(r|b)\)
7:if\(r=1\)then
8: insert rest with duration \(d_{\text{rest}}\sim P(d_{\text{rest}}|b)\)
9:endif
10:endfor
```

**Algorithm 1** Pseudo melody generation

The context melody vector \({\bf n}_{t}\) is a binary feature vector that includes a musical notation type (i.e., note or rest), a duration8, and a pitch for each note/rest in the context window. We collect notes and rests around the target note \(m_{i(t)}\) for the current word position \(t\) with a window size of \(10\) (i.e., \(m_{i(t)-10},...,m_{i(t)},...,m_{i(t)+10}\)).

Footnote 8: We rounded each duration to one of the values 60, 120, 240, 360, 480, 720, 960, 1200, 1440, 1680, 1920, and 3840 and use one-hot encoding for each rounded duration.

For pitch information, we use a gap (pitch interval) between a target note \(m_{i(t)}\) and its previous note \(m_{i(t-1)}\). Here, the pitch is represented by a MIDI note number in the range 0 to 127. For example, the target and its previous notes are 68 and 65, respectively, and the gap is \(+3\).

### Training strategies

PretrainingThe size of our melody-lyrics alignment data is limited. However, we can obtain a large amount of raw lyrics. We, therefore, pretrain the model with 53,181 raw lyrics and then fine-tune it with the melody-lyrics alignment data. In pretraining, all context melody vectors \({\bf n}_{t}\) are zero vectors. We refer to these pretrained and fine-tuned models as _Lyrics-only_ and _Fine-tuned_ models, respectively.

Learning with pseudo-melodyWe propose a method to increase the melody-lyrics alignment data by attaching _pseudo melodies_ to the obtained 53,181 raw lyrics. We refer to the model that uses this data as the _Pseudo-melody_ model.

Algorithm 1 shows the details of pseudo-melody generation. For each syllable in the lyrics, we first assign a note to the syllable by sampling the probability distributions. The pitch of each note is generated based on the trigram probability. Then, we determine whether to generate a rest next to it. Since we established the correlations between rests and boundaries of lyrics in Section 3, the probability for a rest and its duration is conditioned by a boundary

Figure 6: Distribution of the number of boundaries in pseudo-data.

 

type next to the target syllable. All probabilities are calculated using the training split of the melody- lyrics alignment data.

Figure 6 shows the distributions of the number of boundaries in the pseudo data. The distributions closely resemble those of gold data in Figure 4.

## 5 Quantitative evaluation

We evaluate the proposed Melody-conditioned RNNLMs quantitatively based on two evaluation metrics: (1) a test set perplexity for measuring the fluency; (2) a line/block boundary replication task for measuring the consistency between the melody and boundaries in the generated lyrics.

### Experimental setup

In our model, we chose the dimensions of the word embedding vectors and context melody representation vectors to 512 and 256, respectively, and the dimension of the LSTM hidden state was 768. We used a categorical cross-entropy loss for outputs \({\mathbf{y}_{w}^{t}}\) and \({\mathbf{y}_{s}^{t}}\), Adam Kingma and Ba (2014) with an initial learning rate of 0.001 for parameter optimization, and a mini-batch size of 32. We applied an early-stopping strategy with a maximum epoch number of 100, and training was terminated after five epochs of unimproved loss on the validation set. For lyrics generation, we used a beam search with a width of 10. An example of the generated lyrics is shown in the supplemental material.

### Evaluation metrics

PerplexityTest-set perplexity (PPL) is a standard evaluation measure for language models. PPL measures the predictability of wording in original lyrics, where a lower PPL value indicates that the model can generate fluent lyrics. We used PPL and its variant PPL-W, which excludes line/block boundaries, to investigate the predictability of words.

Accuracy of boundary replicationUnder the assumption that the line and block boundaries of the original lyrics are placed at appropriate positions in the melody, we evaluated consistency between the melody and boundaries in the generated lyrics by measuring the reproducibility of the boundaries in the original lyrics. Here the metric we used was \(F_{1}\)-measure of the boundary positions. We also asked a person to place line and block boundaries at plausible positions for randomly selected 10 input melodies that the evaluator has never heard. This person is not a professional musician but an experienced performer educated on musicology. The bottom part of Table 1 represents the human performance.

### Effect of Melody-conditioned RNNLM

To investigate the effect of our language models, we compared the following six models. The first one is (1) a _Lyrics-only_ model, a standard RNNLM trained with 54,081 song lyrics without melody information. The second and third ones are baseline Melody-conditioned RNNLMs where the proposed training strategies are not applied: (2) a _Full-data_ model trained with mixed data (54,081 song lyrics and 900 melody-lyrics alignments of those), and (3) an _Alignment-only_ model trained with only 900 melody-lyrics alignment data. The fourth one is a strong baseline to evaluate the performance of the proposed approaches: (4) a _Heuristic_ model that (i) assigns a line/block boundary to a rest based on its duration with the same probability, as reported in Figure 4, and (ii) fills the space between any two boundaries with lyrics of the appropriate syllable counts. This _Heuristic_ model computes the following word probability:

\[P(w_{l}|w_{0},...,w_{t-1},\textbf{m})= \tag{8}\]

where \(Q\) is the same probability as reported in Figure 4. \(P_{\mathrm{LSTM}}\) is the word probability calculated by a standard LSTM language model. The remaining two are Melody-conditioned RNNLMs with the proposed learning strategies: (5) _Fine-tuned_ and (6) _Pseudo-melody_ models.

\begin{table}
\begin{tabular}{c|r r r r r} \hline  & \multicolumn{2}{c}{Perplexity} & \multicolumn{2}{c}{\(F_{1}\)-measure} \\ Model & PPL & PPL-W & BOB & BOL & UB \\ \hline _Lyrics-only_ & 138.0 & 225.0 & 0.121 & 0.061 & 0.106 \\ _Full-data_ & 135.9 & 222.1 & 0.122 & 0.063 & 0.108 \\ _Aigment-only_ & 173.3 & 314.8 & 0.298 & 0.287 & 0.477 \\ _Heuristic_ & 175.8 & 284.7 & **0.373** & 0.239 & 0.402 \\ _Fine-tuned_ & 152.2 & 275.5 & 0.260 & **0.302** & **0.479** \\ _Pseudo-melody_ & **115.7** & **197.5** & 0.318 & 0.241 & 0.406 \\ \hline (w/o y) & & & & & \\ _Fine-tuned_ & 155.1 & 278.1 & 0.318 & 0.241 & 0.366 \\ _Pseudo-melody_ & 118.0 & 201.5 & 0.312 & 0.250 & 0.406 \\ \hline _Human_ & - & - & 0.717 & 0.671 & 0.751 \\ \hline \end{tabular}
\end{table}
Table 1: Results of the quantitative evaluation. “UB” denotes the score for unlabeled matching of line/block boundaries. “w/o y ” denotes the exclusion of the syllable-count output layer.

 

The top part of Table 1 summarizes the performance of these models. Regarding the boundary replication, the _Heuristic_, _Alignment-only_, _Finetuned_, and _Pseudo-melody_ models achieved higher performance than the _Lyrics-only_ model for unlabeled matching of line/block boundaries (i.e., UB). This result indicates that our Melody-conditioned RNNLMs successfully capture the consistency between melody and boundaries of lyrics. The results of the _Full-data_ model is low (as expected) because the size of the melody-lyrics alignment data is far smaller than that of the raw lyrics data and this harms the learning process of the dependency between melody and lyrics. For the block boundary, the _Heuristic_ model achieved the best performances. For the line boundary, the _Finetuned_ model achieved the best performances.

Regarding PPL and PPL-W, the _Lyrics-only_, _Full-data_, and _Pseudo-melody_ models show better results than the other models. The _Fine-tuned_ model shows reduced performance compared with the _Lyrics-only_ model because fine-tuning with a small amount of data causes overfitting in the language model. Also, the training size of the _Alignment-only_ model is insufficient for learning a language model of lyrics. Interestingly, the _Pseudo-melody_ model achieved better performance than the _Full-data_ model and overall achieved the best score. This result indicates that the _Pseudo-melody_ model uses the information of a given melody to make a better prediction of its lyrics word sequence. On the other hand, the _Heuristic_ model had the worst performance, despite training with a large amount of raw lyrics. We analyze the reason for such performance and describe our results in Section 5.5. It is not necessarily clear which to choose, either the _Fine-tuned_ or _Pseudo-melody_ model, which may depend also on the size and diversity of the training and test data. However, one can conclude at least that combining a limited-scale collection of melody-lyrics alignment data with a far larger collection of lyrics-alone data boosts the model's capability of generating a fluent lyrics which structurally fits well the input melody.

### Effect of predicting syllable-counts

To investigate the effect of predicting syllable-counts, we compared the performance of the proposed models to models that exclude the syllable-count output layer \({\bf y}_{s}\). The middle part of Table 1 summarizes the results. For the pretraining strategy, the use of \({\bf y}_{s}\) successfully alleviates data sparsity when learning the correlation between syllable counts and melodies from only words themselves. As can be seen, the model without \({\bf y}_{s}\) shows reduced performance relative to both PPLs and the boundary replication. On the other hand, for the pseudo-melody strategy, the two models are competitive in both measures. This means that the _Pseudo-melody_ model obtained a sufficient amount of word-melody input pairs to learn the correlation.

### Analysis of melody and generated lyrics

To examine whether the models can capture correlations between rests and boundaries of lyrics, we calculate the proportion of the word, line, and block boundaries in the original lyrics and in the lyrics generated by the _Heuristic_ and _Pseudo-melody_ model for the test set (Figure 7). The proportion of \(\langle\text{BOL}\rangle\) and \(\langle\text{BOB}\rangle\) generated by the _Heuristic_ model are almost equivalent to those of the original lyrics. On the other hand, for the _Pseudo-melody_ model, the proportion of line/block boundary types for the longer rests are smaller than that of the original lyrics.

Although the _Heuristic_ model reproduces the proportion of the original line/block boundaries, the model had a low performance in terms of PPL,

Figure 7: Distribution of the number of boundaries in the test set and lyrics generated by the _Heuristic_ and _Pseudo-melody_ models.

 

as shown in Section 5.3. By investigating the lyrics generated by the _Heuristic_ model, we found that the model tends to generate line/block boundaries after the melody rest, even if the two rests are quite close. Figure 8 shows the distributions of the syllable per line / block frequency and the distributions of the Jensen-Shannon divergence. While the Heuristic model tends to generate short lines/blocks, our model generates the lyrics so that lines/blocks do not become too short. This result supports that (i) our model is trained using melody and lyric contexts and (ii) the heuristic approach, which simply generates line/block boundaries based on the distribution in Figure 4, cannot generate fluent lyrics with well-formed line/block lengths.

## 6 Qualitative evaluation

To asses the quality of the generated lyrics, inspired by Oliveira (2015), we asked 50 Yahoo crowdsourcing workers to answer the following five questions using a five-point Likert scale:

**Listenability (L)** When listening to melody and lyrics, are the positions of words, lines, and segments natural? (1=_Poor_ to 5=_Perfect_)

**Grammaticality (G)** Are the lyrics grammatically correct? (1=_Poor_ to 5=_Perfect_)

**Line-level meaning (LM)** Is each line in the lyrics meaningful? (1=_Unclear_ to 5=_Clear_)

**Document-level meaning (DM)** Are the entire lyrics meaningful? (1=_Unclear_ to 5=_Clear_)

**Overall quality (OQ)** What is the overall quality of the lyrics? (1=_Terrible_ to 5=_Great_)

For the evaluation sets, we randomly selected four melodies from the RWC Music Database Goto et al. (2002). For each melody, we prepared four lyrics generated by the _Heuristic_, _Lyrics-only_, _Fine-tuned_, and _Pseudo-melody_ models. Moreover, to obtain an upper bound for this evaluation, we used the lyrics created by amateur writers: we asked four native Japanese speakers to write lyrics on the evaluation melody. One writer was a junior high school teacher of music who had experience in music composition and writing lyrics. Three writers were graduate students with different levels of musical expertise. Two of the three writers had experience with music composition, but none of them had experience with writing lyrics.9 As a result, we obtained 50 (workers) \(\times\) 4 (melodies) \(\times\) 5 (lyrics) samples in total. We note that workers did not know whether lyrics were created by a human or generated by a computer.

Footnote 9: We release lyrics and audio files used in the qualitative evaluation on the Web ([https://github.com/KentoW/deep-lyrics-examples](https://github.com/KentoW/deep-lyrics-examples)).

Table 2 shows the average scores, standard deviations, and medians for each measure. Regarding the "Listenability" evaluation, workers gave high scores to the _Fine-tuned_ and _Pseudo-melody_ models that are trained using both the melody and lyrics. This result is consistent with the perplexity evaluation result. On the other hand, regarding the "Grammaticality" and "Meaning" evaluation, workers gave high scores to the _Lyrics-only_ and _Pseudo-melody_ models that are well-trained on a large amount of text data. This result is consiste

Figure 8: Distribution of the syllable count of the generated lines/blocks



the boundary replication task. Regarding the "Over-all quality" evaluation, the _Pseudo-melody_ model outperformed all other models. These results indicate our pseudo data learning strategy contributes to generating high-quality lyrics. However, the quality of lyrics automatically generated is still worse than the quality of lyrics that humans produce, and it still remains an open challenge for future research to develop computational models that generate high-quality lyrics.

## 7 Related work

In the literature, a broad range of research efforts has been reported for computationally modeling lyrics-specific properties such as meter, rhythm, rhyme, stress, and accent Greene et al. (2010); Reddy and Knight (2011); Watanabe et al. (2014, 2016). While these studies provide insightful findings on the properties of lyrics, none of those takes the approach of using melody-lyrics parallel data for modeling correlations of lyrics and melody structures. One exception is the work of Nichols et al. (2009), who used melody-lyrics parallel data to investigate, for example, the correlation between syllable stress and pitch; however, their exploration covers only correlations at the prosody level but not structural correlations.

The same trend can be seen also in the literature of automatic lyrics generation, where most studies utilize only lyrics data. Barbieri et al. (2012) and Abe and Ito (2012) propose a model for generating lyrics under a range of constraints provided in terms of rhyme, rhythm, part-of-speech, etc. Potash et al. (2015) proposes an RNNLM that generates rhymed lyrics under the assumption that rhymes tend to coincide with the end of lines. In those studies, the melody is considered only indirectly; namely, input prosodic/linguistic constraints/preferences on lyrics are assumed to be manually provided by a human user because the proposed models are not capable of interpreting and transforming a given melody to constraints/preferences.

For generating lyrics for a given melody, we have so far found in the literature two studies which propose a method. Oliveira et al. (2007) and Oliveira (2015) manually analyze correlations among melodies, beats, and syllables using 42 Portuguese songs and propose a set of heuristic rules for lyrics generation. Ramakrishnan A et al. (2009) attempt to induce a statistical model for generating melodic Tamil lyrics from melody-lyrics parallel data using only ten songs. However, the former captures only phonological aspects of melody-lyrics correlations and can generate a small fragment of lyrics (not an entire lyrics) for a given piece of melody. The latter suffers from the severe shortage of data and fails to conduct empirical experiments.

## 8 Conclusion and future work

This paper has presented a novel data-driven approach for building a melody-conditioned lyrics language model. We created a 1,000-song melody-lyrics alignment dataset and conducted a quantitative investigation into the correlations between melodies and segment boundaries of lyrics. No prior work has ever conducted such a quantitative analysis of melody-lyrics correlations with this size of data. We have also proposed a RNN-based, melody-conditioned language model that generates fluent lyrics whose word/line/block boundaries fit a given input melody. Our experimental results have shown that: (1) our Melody-conditioned RNNLMs capture the consistency between melody and boundaries of lyrics while maintaining word fluency; (2) combining a limited-scale collection of melody-lyrics alignment data with a far larger collection of lyrics-alone data for training the model boosts the model's competence; (3) we have also produced positive empirical evidence for the effect of applying a multi-task learning schema where the model is trained for syllable count prediction as well as for word prediction; and (4) the human judgments collected via crowdsourcing showed that our model improves the quality of generated lyrics.

For future directions, we plan to further extend the proposed model for capturing other aspects of lyrics/melody discourse structure such as repetitions, verse-bridge-chorus structure, and topical coherence of discourse segment. The proposed method for creating melody-lyrics alignment data enables us to explore such a broad range of aspects of melody-lyrics correlations.

### Acknowledgments

This study utilized the RWC Music Database (Popular Music). This work was partially supported by a Grant-in-Aid for JSPS Research Fellow Grant Number JP16J05945, JSPS KAKENHI Grant Numbers JP15H01702, and JST ACCEL Grant Number JPMJAC1602. The authors would like to thank Dr. Paul Reisert for the English language review 



## References

* Abe and Ito (2012) Chihiro Abe and Akinori Ito. 2012. A Japanese lyrics writing support system for amateur songwriters. In _Proceedings of Asia-Pacific Signal & Information Processing Association Annual Summit and Conference 2012 (APSIPA ASC 2012)_. pages 1-4.
* Austin et al. (2010) Dave Austin, Jim Peterik, and Cathy Lynn Austin. 2010. _Songwriting for Dummies_. Wileys.
* Barbieri et al. (2012) Gabriele Barbieri, Francois Pachet, Pierre Roy, and Mirko Degli Esposti. 2012. Markov constraints for generating lyrics with style. In _Proceedings of the 20th European Conference on Artificial Intelligence (ECAI 2012)_. pages 115-120.
* Goto et al. (2002) Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuich Oka. 2002. RWC Music Database: Popular, classical and jazz music databases. In _Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR 2002)_. volume 2, pages 287-288.
* Greene et al. (2010) Erica Greene, Tugba Bodrumlu, and Kevin Knight. 2010. Automatic analysis of rhythmic poetry with applications to generation and translation. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP 2010)_. pages 524-533.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _Proceedings of the 32nd International Conference on Machine Learning (ICML 2015)_. volume 37, pages 448-456.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint_ arXiv:1412.6980.
* Kudo et al. (2004) Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP 2004)_. pages 230-237.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _Proceedings of Interspeech 2010_. pages 1045-1048.
* Needleman and Wunsch (1970) Saul B. Needleman and Christian D. Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. _Journal of Molecular Biology_ 48(3):443-453.
* Nichols et al. (2009) Eric Nichols, Dan Morris, Sumit Basu, and Christopher Raphael. 2009. Relationships between lyrics and melody in popular music. In _Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR 2009)_. pages 471-476.
* Oliveira (2015) Hugo Goncalao Oliveira. 2015. Tra-la-lyrics 2.0: Automatic generation of song lyrics on a semantic domain. _Journal of Artificial General Intelligence_ 6(1):87-110.
* Goncalo Oliveira et al. (2007) Hugo R. Goncalo Oliveira, F. Amialcar Cardoso, and Francisco C. Pereira. 2007. Tra-la-lyrics: An approach to generate text based on rhythm. In _Proceedings of the 4th International Joint Workshop on Computational Creativity_. pages 47-55.
* Potash et al. (2015) Peter Potash, Alexey Romanov, and Anna Rumshisky. 2015. GhostWriter: Using an LSTM for automatic Rap lyric generation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)_. pages 1919-1924.
* Ramakrishnan A et al. (2009)Ananth Ramakrishnan A, Sankar Kuppan, and Sobha Lalitha Devi. 2009. Automatic generation of Tamil lyrics for melodies. In _Proceedings of the Workshop on Computational Approaches to Linguistic Creativity_. pages 40-46.
* Reddy and Knight (2011) Sarvana Reddy and Kevin Knight. 2011. Unsupervised discovery of rhyme schemes. In _Proceeds of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2011)_. pages 77-82.
* Ueda (2010) Tatsuji Ueda. 2010. \(\chi<\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>\chi>>\chi>\chi>\chi>\chi>\chi>>\chi>\chi>\chi>\chi>>\chi>\chi>\chi>>\chi>\chi>\chi>>\chi>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>\chi>>\chi>>\chi>\chi>>\chi>>\chi>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>\chi>>>\chi>>\chi>>>\chi>>\chi>>>\chi>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>\chi>>>>\chi>>