IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1167

PAPER

Modeling Storylines in Lyrics
Kento WATANABE†a) , Yuichiroh MATSUBAYASHI† , Nonmembers, Kentaro INUI†,†† ,Satoru FUKAYAMA††† , Members, Tomoyasu NAKANO††† , Nonmember, and Masataka GOTO††† , Member

SUMMARY
This paper addresses the issue of modeling the discourse
nature of lyrics and presented the first study aiming at capturing the two
common discourse-related notions: storylines and themes. We assume thata storyline is a chain of transitions over topics of segments and a song has
at least one entire theme. We then hypothesize that transitions over topics of lyric segments can be captured by a probabilistic topic model which
incorporates a distribution over transitions of latent topics and that such a
distribution of topic transitions is aﬀected by the theme of lyrics. Aiming totest those hypotheses, this study conducts experiments on the word prediction and segment order prediction tasks exploiting a large-scale corpus of
popular music lyrics for both English and Japanese (around 100 thousandsongs). The findings we gained from these experiments can be summarized
into two respects. First, the models with topic transitions significantly outperformed the model without topic transitions in word prediction. This
result indicates that typical storylines included in our lyrics datasets wereeﬀectively captured as a probabilistic distribution of transitions over latent
topics of segments. Second, the model incorporating a latent theme variable
on top of topic transitions outperformed the models without such variables
in both word prediction and segment order prediction. From this result, we
can conclude that considering the notion of theme does contribute to themodeling of storylines of lyrics.
key words: bayesian model, generative model, lyrics structure, lyrics understanding, natural language processing

1.

Introduction

Lyrics are an important element in popular music that conveys stories and expresses emotion. Unlike prose text, lyrics
are created in consideration of its specific properties such as
fitness to rhythm and melody, and rhetoric with rhyme, refrain and repetition [1], [2]. In writing a piece of lyrics fora given piece of music, the writer should select right words
such that their syllables fit the rhythm or melody of the music. The writer may also consider using rhymes, refrains orrepetitions to color the entire story rhetorically as in the example lyrics shown in Fig. 1, where rhymes can be seen atnight, light and tight in Segment 2). Writing lyrics is thus acomplex task.
These characteristics of lyrics have been motivating a
range of research for computer-based modeling of lyrics andManuscript received June 10, 2017.
Manuscript revised November 28, 2017.
Manuscript publicized December 22, 2017.
†
The authors are with Tohoku University, Sendai-shi, 980–
8579 Japan.
††
The author is with RIKEN Center for Advanced Intelligence
Project, Tokyo, 103–0027 Japan.
†††
The authors are with National Institute of Advanced Industrial
Science and Technology (AIST), Tsukuba-shi, 305–8568 Japan.
a) E-mail: kento.w@ecei.tohoku.ac.jp
DOI: 10.1587/transinf.2017EDP7188

Fig. 1 Lyrics with a storyline (title: Don’t Say Good bye (RWC-MDBP-2001 No. 90 from RWC Music Database [17])).

computer-assisted or fully-automated creation of lyrics [3]–[10]. In particular, building a computational model of lyrics
is an important research goal. Once a reasonably sophisticated computational model of lyrics is obtained, the model
will provide us a better understanding of the nature andstructure of lyrics, which will then allow us to consider
building computer systems which can enhance the creativity of human lyrics writers. In reality, however, while an increasing number of papers have been published for demonstrating computer systems that automatically generate lyrics
or assist human lyricists [3]–[10], research for modelinglyrics and understanding their properties is still limited [11]–[15].
One crucial issue we miss in previous studies is modeling the nature of lyrics as discourse. Similar to prose text,
a piece of lyrics typically comprises discourse segments;
namely, lyrics of popular music typically has verse, bridgeand chorus segments [16] and such segments may comprise
more fine-grained segments as in Fig. 1. Each segment provides part of the entire story and the segments are organized(or sequentially ordered) so as to constitute a coherent structure as a whole. In spite of its importance, however, noprior study has ever addressed the issue of modeling this
discourse-oriented nature of lyrics.
Motivated by this background, in this paper, we reporton our novel study for building a computational model of the
discourse nature of lyrics. We focus on two notions whichcharacterize lyrical discourse: storyline and theme. Bothnotions are described in textbooks on lyrics writing [1], [2].
A segment of lyrics is assumed to have its own pur-c 2018 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1168

Fig. 2The notion of storylines and themes in lyrics.

pose, which corresponds to a discourse segment purposein terms of discourse analysis research [18]. In Fig. 1, forexample, Segment 1 introduces the story, Segment 2 retrospects a past event, and Segment 3 expresses an emotion
which arises from the retrospection. We model a storyline
as such a chain of coherent shifts between discourse segment
purposes. Specifically, we capture typical types of discourse
segment purposes as latent topics by applying topic modeling techniques [19] to a large collection of lyrical texts,
and then model typical storylines of lyrics as a probability
distribution over the transition of latent topics over successive segments (Fig. 2). On top of storylines, we additionallyconsider the notion of theme, which we assume to be an
entire discourse purpose. We assume that each song has atleast one theme and each theme aﬀects the distribution overboth topic transitions and word choices. For the lyrics in
Fig. 1, for example, our model provides a result with which
we can understand its theme as “Sweet Love” and estimates
the theme-sensitive distributions over topic transitions andword choices.
In order to examine how well our model of lyrics fit
real-world data, we experiment with two distinct prediction
tasks, word prediction and segment order prediction, andcompare four variant models with diﬀerent settings for considering storylines and themes. In the experiments, the models were trained with unsupervised learning over a largescale corpus of popular music lyrics for both English andJapanese (around 100 thousand songs). The results demonstrate that the consideration of storylines (topic transitions)and themes contributes to improved prediction performance.
In what follows, we review related work in Sect. 2 anddescribe our novel method of modeling lyrics in Sect. 3. We
then present our experiments in Sect. 4 before concluding
this study in Sect. 5.
2.

Related Work

2.1

Modeling Structure of Lyrics

Plenty of studies for capturing lyric-specific properties havebeen reported, where a broad range of music elements including meter, rhythm, rhyme, stressed/unstressed syllables,
and accent are studied. Reddy and Knight [14] developed alanguage-independent rhyme model based on a Markov process that finds rhyme schemes. Greene et al. [13] employeda finite-state transducer to assign syllable-stress pattern toall words in each line. Nichols et al. [12] identified several
patterns in the relationship between the lyrics and melody inpopular music by measuring the correlation between textual
salience and musical salience. Mayer et al. [11] trained a
support vector machine to classify music genres using only
textual features such as rhyme and part-of-speech patterns.
Barbieri et al. [5], Abe and Ito [6], and Ramakrishnan andDevi [4] generated English, Japanese and Tamil lyrics thatsatisfy a given input constraint, such as rhyme, rhythm, andpart-of-speech templates. Wu et al. [7] applied stochastictransduction grammar induction algorithms to generate fluent rhyming hip hop lyrics. Watanabe et al. [15] proposed
a computational model that predicts segment boundaries
in lyrics by utilizing a self-similarity matrix, which is frequently used in audio-based music structure analysis.
Several studies aim at modeling semantic structure of
lyrics. Kleedorfer et al. [20] classified lyrics according to
topical clusters calculated using nonnegative matrix factorization [21]. Sasaki et al. [22] visualized lyric clusters calculated using Latent Dirichlet Allocation (LDA) [19].
These prior studies share the motivation of modelinglyric-specific structure with our study. However, they havenot considered capturing the discourse-oriented nature of
lyrics whereas our study aims at modeling storylines (i.e.
coherent transitions of discourse segment purposes) andthemes (i.e. entire discourse purposes) of lyrics.
2.2 Modeling Topic Sequence
Transition of topics has been studied in the context of topic
modeling for sequential text data such as newspapers, weblogs, and conversations. Iwata et al. [23] proposed a Topic
Tracking Model (TTM), an extension of LDA, to models
topic sequences. In the TTM, the topic distribution changes
at each time. The TTM analyzes changes in user interest (e.g., interest in weblogs and microblogs). Blei andLaﬀerty [24] proposed the Dynamic Topic Model (DTM),
which is similar to the TTM. In the DTM, the prior distribution of topic distribution changes at each time. The DTM
analyzes changes in topic over time (e.g., topics in news articles and academic papers). The TTM and DTM have a
topic distribution for a specific date (e.g., the DTM can train
the topic distribution in a given period). Although the DTM
and TTM can represent the topic sequence, extending thesemodel to lyrics is diﬃcult because, in lyrics, a segment’s
topic is time-independent.
Barzilay et al. proposed Content Model (CM), which
is typically used for discourse analysis [25], to model topic
sequences in documents without date information. CMs aresentence-level hidden Markov models that capture the sequential topic structure of a news event, such as earthquakes.

WATANABE et al.: MODELING STORYLINES IN LYRICS

1169

Fig. 3 Plate notation of base models (a, b) and the proposed combination models (c, d). The shaded
nodes are observed variables, dots are hyperparameters, x, y, and z are latent variables, and ψ, θ, φ, andλ are probability distributions.

Several studies extended Barzilay’s model to dialog acts
(e.g., questions and responses) [26], [27]. Ritter et al. [26]assumed that an observed sentence is generated from eithera dialog act-specific language model (e.g., questions and responses) or a dialog-specific language model (e.g., food andmusic). Zhai and Williams [27] assumed that an observed
word is generated from either a CM or an LDA and modeled
the latent structure in task-oriented dialog. In their study,
the sequential structure of dialog is modeled as a transition
distribution.
We share the core concept as these studies and apply a
CM to lyrics to model storylines (See Sect. 3.2). We then extend the CM to capture theme and investigate the eﬀects ofconsidering themes on top of storylines in our experiments.
3.

Model Construction

Our final objective is to model the storyline of lyrics. However, precise modeling and representation of storylines remain an open issue. As mentioned previously, lyricists consider the order of topics when creating storylines; if the order changes, the content of the lyrics also changes. Therefore, we assume that a better storyline model can be used to
predict the order of segments and the words in lyrics.
Based on the above assumption, we explore diﬀerent
topic sequence models to improve prediction performance.
Lyricists often consider the order of topics when they create
storylines; therefore, we assume that topic sequences can be
represented as a probabilistic distribution of transitions overlatent states. Since lyricists often assign a certain role toeach segment, we assume that the segment is in one latentvariable for a given lyrical content and words are derived
from each latent state. Moreover, we assume that lyrics
in a song are in one latent variable (i.e., a theme) because
lyricists often create storylines according to themes. Basedon the above idea, we prepared four data-driven Bayesianmodels. By comparing the performance of the models, weinvestigate which encoding method can better model thestoryline.
In the following, we first describe the notations used inthis study and two baseline methods for modeling the storyline. Finally, we propose two extended combination models
to handle theme and storyline simultaneously.
3.1 Preliminaries
We assume that we have a set of M lyrics (songs). The
lyric is an index between 1 and M, where M is the number of songs. The m-th lyric contains Sm segments andhas a single theme denoted as the latent variable ym . The
theme is an index between 1 and I, where I is the number of themes. The s-th segment contains a bag of words
denoted as {wm,s,1 , wm,s,2 , . . . , wm,s,Nm,s }, where wm,s,n is an index between 1 and V, where V is the vocabulary size. Nm,sis the number of words in the s-th segment of the m-th lyric.
In addition, each segment has a single topic denoted as thelatent variable zm,s . The topic is an index between 1 andJ, where J is the number of topics. The storyline is represented as the sequence of a segment’s topic denoted aszm = zm,1 , zm,2 , . . . , zm,Sm .
3.2 Base Model 1: Content Model
We use the CM [25] as a baseline model for the storylineof lyrics because this model is the simplest topic transition model that satisfies our assumption that the topic sequence can be encoded as probabilistic latent state transition. As shown in Fig. 3 (a), we assume that the storyline can be generated from a topic transition distribution
θz . For the s-th segment in the m-th lyric, each topic zm,sis generated from the previous topic zm,s−1 via the transition
probability P(zm,s |θzm,s−1 ). This probability is calculated by

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1170

J-dimensional multinomial distribution θz drawn from
Dirichlet distributions with symmetric hyperparameter α.
Then the word wm,s,n in each segment is generated
from each topic zm,s via topic-specific generative probability P(wm,s,n |φzm,s ). This probability is calculated by
V-dimensional multinomial distribution φz drawn from
Dirichlet distributions with symmetric hyperparameter β.
3.3Base Model 2: Mixture of Segment Model

To investigate the eﬀects of capturing topic transitions, we
also build a model that removes topic transitions from the
CM (Fig. 3 (b)). We refer to this model as the Mixtureof Segment Model (MSM). In the MSM, each segment’s
topic zm,s is generated via the probability without transition
P(zm,s |θ). This probability is calculated by J-dimensional
multinomial distribution θ drawn from Dirichlet distributions with symmetric hyperparameter α. In other words, theMSM has only one probability distribution θ while the CM
has J probability distributions θz .
3.4

Proposed Model 1: Mixture of Unigram and Content
Model

To verify that both theme and topic transition are useful formodeling the storyline, we propose a model that combines
the theme and the topic transition simultaneously, and wecompare this model to the baseline models. The idea behind
this combined modeling is that we can mix a theme-specificmodel and the topic transition model (i.e., the CM) using
linear interpolation assuming that words in lyrics are dependent on both the theme and the topic.
We use the Mixture of Unigram Model (MUM) [28] as
the theme-specific model because it is the simplest modelthat satisfies our assumption; lyrics in a song are in a single
latent variables (i.e., the theme). The MUM assumes thattheme ym is drawn from an I-dimensional theme distribution
ψ and all words in the lyrics are drawn from V-dimensional
multinomial distribution φym as shown in Fig. 3 (c).
In the proposed MUM-CM, we define a binary variable
xm,s,n that uses either the MUM or the CM when the word
wm,s,n is generated. Here, if xm,s,n = 0, the word is drawn
from the MUM’s word distribution φy , and if xm,s,n = 1, theword is drawn from the CM’s word distribution φz . The binary variable x is drawn from a Bernoulli distribution λm
drawn from a beta distribution with symmetric hyperparameter η. In other words, the words depend on both theme andtopic, and the MUM and CM are defined independently inthis model.
Figure 3 (c) shows the plate notation of the MUM-CM.
The generation process in the MUM-CM is as follows.
1. Draw a theme distribution ψ ∼ Dir( )2. For each theme y = 1, 2, . . . , I:
• Draw a distribution of theme words φy ∼ Dir(ζ)
3. For each topic z = 1, 2, . . . , J:

• Draw a topic transition distribution θz ∼ Dir(α)• Draw a distribution of topic words φz ∼ Dir(β)
4. For each lyric m = 1, 2, . . . , M:
• Draw a theme ym ∼ Multi(ψ)• Draw a distribution of binary variableλm ∼ Beta(η)• For each segment s = 1, 2, . . . , Sm :– Draw a topic zm,s ∼ Multi(θzm,s−1 )– For the n-th word wm,s,n in segment s:∗ Draw a binary variable
xm,s,n ∼ Bernoulli(λm )∗ If xm,s,n = 0:
· Draw a word wm,s,n ∼ Multi(φym )∗ If xm,s,n = 1:
· Draw a word wm,s,n ∼ Multi(φzm,s )Here, α, β, , and ζ are the symmetric hyperparameters of
the Dirichlet distribution and η is the symmetric hyperparameter of the beta distribution. The generation probabilityof the m-th lyric is calculated as follows:
P(m) = P(x = 0|λm )

I 

y=1

+ P(x = 1|λm )Sm zall s=1

P(y|ψ)

Nm,s
Sm 


P(wm,s,n |φy )

s=1 n=1

P(z s |θzs−1 )Nm,sP(wm,s,n |φzs )



(1)

n=1

where zall denotes all possible topic sequences. If s = 1, θz0denotes the initial state probabilities. This equation represents that a word wm,s,n is generated from the MUM according to P(x = 0|λm ) or is generated from the CM according
to P(x = 1|λm ).
We use collapsed Gibbs sampling for model inferencein the MUM-CM. For a lyric m, we present the conditional
probability of theme ym for sampling:P(ym = i|y¬m , w, , ζ) ∝ P(ym = i|y¬m , )·P(wm |w¬m , ym = i, y¬m , ζ)(2)where y¬m denotes the topic set except the m-th lyric, w denotes the word set in the training corpus, wm denotes theword set in the m-th lyric, and w¬m denotes the word set in
the training corpus except wm .
We sample topic zm,s for a segment s of lyric m according to the following transition distribution:P(zm,s = j|z¬(m,s) , w, α, β) ∝ P(zm,s = j|z¬(m,s) , α)·P(wm,s |w¬(m,s) , zm,s = j, z¬(m,s) , β)(3)where z¬(m,s) denotes the topic set except the s-th segment in
the m-th lyric, wm,s denotes the word set in the s-th segmentof the m-th lyric, and w¬(m,s) denotes the word set in the
training corpus except wm,s .
For the n-th word in the segment s in the m-th lyric,
we present the conditional probability of its binary variables

WATANABE et al.: MODELING STORYLINES IN LYRICS

1171

Algorithm 1 Model inference for the MUM-CM

Algorithm 2 Model inference for the MCM

1: Initialize parameters in the MUM-CM
2: for each iteration do3:
for each lyrics m in the corpus do
4:
sample ym according to (2)5:for each segment s in m do
6:
sample zm,s according to (3)7:for each word w in s do
8:
sample x according to (4)9:end for10:
end for11:
end for12:
update hyperparameters by using fixed point iteration
13: end for

1: Initialize parameters in the MCM
2: for each iteration do3:
for each lyrics m in the corpus do
4:
sample ym according to (6)5:sample zm according to (7) by FFBS
6:
end for7:
update hyperparameters by using fixed point iteration
8: end for∗ Draw a word wm,s,n ∼ Multi(φzm,s )The generation probability of lyric m is calculated as follows:

xm,s,n for sampling:P(xm,s,n = k|x¬(m,s,n) , w, η, ζ, β)∝ P(xm,s,n = k|x¬(m,s,n) , η)·P(wm,s,n |w¬(m,s,n) , xm,s,n = k, x¬(m,s,n) , ζ, β)P(m) =(4)

Proposed Model 2: Mixture of Content ModelIn the MUM-CM, we assume that theme and storyline
are generated independently. On the other hand, as mentioned in Sect. 1, lyricists often create storylines according
to themes. Therefore, here, we propose the Mixture of Content Model (MCM) to verify this intuition. In the MCM,
when a theme y is generated, a storyline is generated using
the theme-specific topic transition distribution θy,z .
Figure 3 (d) shows the plate notation of the MCM. The
MCM generation process is as follows.
1. Draw a theme distribution ψ ∼ Dir( )2. For each topic z = 1, 2, . . . , J:
• Draw a word distribution φz ∼ Dir(β)• For each theme y = 1, 2, . . . , I:
– Draw a topic distribution θy,z ∼ Dir(α)
3. For each lyric m = 1, 2, . . . , M:
• Draw a theme ym ∼ Multi(ψ)• For each segment s = 1, 2, . . . , Sm :– Draw a topic zm,s ∼ Multi(θym ,zm,s−1 )– For n-th word wm,s,n in segment s:P(y|ψ)

y=1

where x¬(m,s,n) denotes the binary variable set except xm,s,n .
Note that the value of k is always 0 or 1.
We estimate hyperparameters α, β, , ζ, and η using
fixed point iteration [29]. For each sampling iteration, thelatent variables x, y, and z are sampled. Then, new hyperparameters are estimated such that the joint probabilities P(w, y| , ζ), P(w, z|α, β), and P(w, x|η) are maximized,
where y, z, and x denote the latent variable sets in the training corpus.
In summary, the model and parameter inference for the
MUM-CM is shown in Algorithm 1, and the update equations for Gibbs sampling are given in Appendix A.
3.5

I 


Sm 

zall s=1

P(z s |θy,zs−1 )

N
m,s,n


P(wm,s,n |φzs )



(5)

n=1

where zall denotes all possible topic sequences. If z = 1, θy,z0
denotes the initial state probabilities. In this model, P(y|ψ)
represents the mixture ratio of the CMs.
We use collapsed Gibbs sampling for model inferencein the MCM. For the m-th lyric, we present the conditional
probability of theme ym for sampling:P(ym = i|y¬m , z, α, ) ∝ P(ym = i|y¬m , )·P(zm |z¬m , ym = i, y¬m , α)(6)where z denotes the topic set in the training corpus, zm denotes the topic sequence of lyric m (i.e., zm,1 , zm,2 , . . . , zm,Sm ),and z¬m denotes the topic set in the training corpus exceptzm .
In the MCM, topic sequence zm depends on theme ym ,as shown in Fig. 3 (d). Therefore, when a new theme y issampled, the MCM must resample all topic sequences inlyric m simultaneously. To sample topic sequence zm , wepresent the following conditional probability:
P(zm |z¬m , y, w, α, β) ∝ P(zm |z¬m , y, α)·P(wm |w¬m , zm , z¬m , β)(7)However, enumerating all possible topic sequences is infeasible; thus, we use a Forward Filtering-Backward Sampling (FFBS) method [30] that can sample all latent statesin a first-order Markov sequence using dynamic programming. In the FFBS method, the marginal probabilities of a
topic sequence are calculated in the forward filtering step.
Then, topics are sampled from the obtained probabilities inthe backward sampling step. The hyperparameters α, β, andare estimated using fixed point iteration [29].
In summary, the model and parameter inference for the
MCM is shown in Algorithm 2, and the update equations for
Gibbs sampling are given in Appendix B.
4.

Experiments

Here, we examine the eﬀectiveness of the proposed models.
First, we verify that topic transitions are useful for modeling storyline by evaluating the word prediction performance

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1172

Fig. 4 Log-likelihood on English test data under diﬀerent segment topic
settings (the number of themes I is fixed at 30).

Fig. 5 Log-likelihood on Japanese test data under diﬀerent segment topic
settings (the number of themes I is fixed at 30).

among diﬀerent models. We then verify that the storyline
correlates with the theme performing a segment order prediction task. Finally, we evaluate the proposed models qualitatively by exploring the trained topic transition diagrams.
In our experiments, we originally created two large
datasets that contain English and Japanese lyrics of existing songs. One issue that needed to be addressed prior to
conducting the experiments was that no existing lyric corpora annotate musical structure (e.g., verse-bridge-chorustags). In this paper, we assume that segment boundaries areindicated by empty lines inserted by lyricists. In addition,
we assume that lyrics with storylines are divided into 6 to
18 segments. The resulting datasets include 80777 lyrics inthe English dataset and 16563 lyrics in the Japanese dataset.
We randomly split each dataset into 60-20-20% divisions toconstruct the training, development, and test data.
We trained English-only and Japanese-only models.
The collapsed Gibbs sampling ran for 1000 iterations, andthe hyperparameters were updated for each Gibbs iteration.
For training, we only used content words (nouns, verbs, andadjectives) because we assume that the theme and storyline
can be represented using content words.
To extract content words, we use Stanford CoreNLP
for English words [31] and the MeCab part-of-speech parser
for Japanese words [32].

can be seen, the CM outperforms the MSM, which indicatesthat typical storylines were eﬀectively captured as the probabilistic distribution of transition over latent topics of segments. Note that the proposed MUM-CM achieves the best
performance, which indicates that a better storyline model
can be constructed by assuming that the words in lyrics are
generated from both theme and topic. The MCM, however,demonstrates only comparable performance to the CM despite that the MCM has a richer parameter space of topic
transition distributions.

4.1

Word Prediction Task

To verify that the topic transition and theme are useful properties for storyline modeling, we performed a word prediction task, which measures the test set generation probability.
We assume that a better prediction model can capture thestoryline of lyrics more eﬀectively. In this experiment, we
fixed the number of themes to 30 and computed the test set
log-likelihood over the number of segment topics to compare diﬀerent models.
Figures 4 and 5 show the English and Japanese test set
log-likelihood under diﬀerent segment topic settings. As

4.2 Segment Order Prediction Task
In this section, we verify that storyline correlates withtheme. Here, we use the order test metric [33], which isused to measure the predictive power of the sequential structure [26], [27]. With the test order metric, the model predicts a reference segment order from all possible segment
orders. However, enumerating all possible orders is infeasible; thus, we use the approximation method proposed by
Zhai et al. [27]:1. Select N permutations randomly from test data except
reference order A.
2. Calculate the N + 1 document generative probabilities
P(m) whose order is A or N permutations.
3. Choose the hypothesis order A whose generative probability is the best value in the N + 1 orders.
4. Compare the hypothesis order A with the reference order A to calculate Kendall’s tau:τ=c+ (A, A ) − c− (A, A )T (T − 1)/2

(8)

where c+ (A, A ) denotes the number of correct pairwise orders, c− (A, A ) denotes the number of incorrect pairwise orders, and T denotes the number of segments in a lyric. Here,N = 50. This metric ranges from +1 to −1, where +1 indicates that the model selects the reference order and −1 indi-

WATANABE et al.: MODELING STORYLINES IN LYRICS

1173
Table 1
Data
EnglishlyricsJapaneselyricsParameter tuning results with the development set.
Model
Base Model 1: CM
Proposed Model 1: MUM-CM
Proposed Model 2: MCM
Base Model 1: CM
Proposed Model 1: MUM-CM
Proposed Model 2: MCM

I: # of themes
none
120
70
none
30
50

J: # of topics
30
4
11
15
8
7

Fig. 7 Average Kendall’s τ for Japanese lyrics against the number of random permutations (the vertical range depicts the confidence intervals of thehuman assessment results).

Fig. 6 Average Kendall’s τ for English lyrics against the number of random permutations.

cates that model selects the reverse order. In other words, ahigher value indicates that the sequential structure has beenmodeled successfully.
To tune the best parameters (i.e., the number of themes
I and number of topics J), we use a grid search on thedevelopment set. Table 1 shows the parameters for eachmodel that achieve the best segment order prediction task
performance.
As a lower bound baseline, we use a model that randomly selects a hypothesis order A (i.e., this lower boundis equivalent to the performance of Base Model 2 that does
not handle topic transition). To obtain an upper bound forthis task, nine Japanese evaluators selected the most plausible order from six orders that include a reference order.
Here, N = 5 for the human assessments due to cognitive
limitations relative to the number of orders. In this manual evaluation, each evaluator randomly selected unknown
lyrics. As a result, we obtained 93 orders.
Figures 6 and 7 show Kendall’s tau averaged over all
English and Japanese test data, respectively. The verticalrange shows 95% confidence intervals for the human assessment results. The experimental results indicate that,compared to the lower bound, the proposed models that
handle topic transition and theme (i.e., the MUM-CM andMCM) have the predictive power of the sequential structure.
This result shows that topic transition and theme are useful properties for storyline modeling. The proposed MCM
outperformed all other models on both test sets, while the
MUM-CM only demonstrated performance comparable tothat of the CM. We also conducted analysis of variance(ANOVA) followed by post-hoc Tukey tests to investigate

the diﬀerences among these models (p < 0.05), drawing theconclusion that the diﬀerence between the MCM and theother models is statistically significant. These results showthat storyline in lyrics correlates to theme. In contrast to theword prediction task, the MUM-CM has a similar predictive
performance as the CM because the MUM-CM has only one
topic transition distribution to model the order of segments,
which is also the case for the CM.
For Japanese lyrics with N = 5, Fig. 7 shows that
Kendall’s tau for the human evaluation was 0.58 ± 0.11,
while the best performance of the model was 0.35. To investigate the cause of this diﬀerence, we asked the evaluators
to write comments on this task. We found that most evaluators selected a single order by considering the following
tendencies.
• Chorus segments tend to be the most representative,
uplifting, and thematic segments. For example, thechorus often contains interlude words, such as “hey”
and “yo”, and frequently includes the lyrical message,
such as “I love you”. Moreover, the chorus is often thefirst or last segment; therefore, evaluators tend to first
guess which segment is the chorus.
• Verse segments tend to repeat less frequently than choruses.
The human annotators were able to take these factors
into account whereas the proposed models cannot consider
verse-bridge-chorus structure. This issue could be addressedby combining the storyline of lyrics with the musical structure. We believe this direction will open an intriguing new
field for future exploration.
4.3 Analysis of Trained Topic Transitions
Our experimental results indicate that topic transition andtheme are useful properties for modeling a storyline. Thus,
we are interested in understanding what kinds of themes andtopic transitions our model can acquire. Here, to interpret

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1174
Table 2 Representative words of each topic for English lyrics in MCM@I = 70, J = 11. The topic
label indicates our arbitrary interpretation of the representative words.
z
1

Label
Abbreviation

2

Spanish3

Exciting

4

Religious

5

Love

6

Explicit

7

Locomotion

8

Interlude

9

Feeling

10

Love

11

Life

Representative words in each topic (top 40 words from P(w|φ s ))ah, mi, dem, di, yuh, man, nah, nuh, gal, fus, work, inna, woman, pon, gim, fi, dat, seh, big, mek, weh, u, jump, wah, deh, yah,wid, tek, jah, waan, wine, red, !!!, youth, Babylon, ghetto, neva, hurry, l, nuﬀ
que, de, tu, el, te, lo, se, yo, un, e, si, por, con, como, amor, una, ti, le, quiero, para, sin, mas, esta, pa, pero, todo, al, solo, las,cuando, hay, voy, corazon, che, soy, je, los, del, vida, tengo
like, hey, dance, uh, ya, right, body, party, put, shake, move, hand, hot, everybody, boy, beat, floor, c’mon, play, show, ’em, club,bang, drop, huh, lady, bounce, clap, sexy, freak, check, pop, push, low, top, shawty, boom, step, hip, dj
come, day, sing, god, song, lord, hear, Christmas, call, bring, child, new, heaven, beautiful, well, king, name, Jesus, pray, soul,angel, wish, yes, help, year, bear, happy, people, joy, old, son, Mary, bell, peace, father, mother, ring, holy, praise, voice
love, feel, need, heart, hold, give, fall, night, dream, world, eye, light, tonight, shine, little, rain, fly, sun, touch, inside, fire, sky,
kiss, free, sweet, star, cry, burn, true, close, mine, arm, alive, set, tear, somebody, open, higher, deep, blue
nigga, shit, fuck, bitch, cause, money, niggaz, ass, hit, real, y’, wit, hoe, game, street, em, bout, fuckin, gettin, rap, gun, blow,hood, kid, pay, damn, catch, block, tryin, aint, thug, motherfucker, dick, smoke, straight, house, g, talkin, dog, buy
go, get, let, back, ta, take, keep, home, round, turn, run, rock, ride, long, stop, roll, ready, got, road, high, slow, far, music, train,start, town, goin, please, drive, control, radio, fight, fast, car, city, ground, rollin, foot, comin, outta
oh, la, yeah, ooh, da, whoa, ba, ha, doo, woah, yea, ay, ho, ohh, oooh, mmm, ooo, woo, hoo, oo, dum, ohhh, oh-oh, ahh, ooooh,
oooo, wee, la., ohhhh, click, dee, fa, bop, shame, l.a., hmmm, ahhh, drip, trouble, mm
know, say, time, never, see, make, one, way, think, life, thing, try, find, leave, look, nothing, always, everything, believe, change,lose, live, mind, much, something, wait, better, ’cause, break, wrong, lie, hard, end, word, stay, mean, seem, friend, someone, carena, wan, gon, baby, girl, want, tell, good, bad, alright, talk, crazy, nobody, cuz, im, ai, babe, bye, dont, lovin, fine, feelin, worry,
pretty, phone, nothin, fun, thinkin, guy, cos, kind, spend, doin, next, number, sex, treat, cool, honey, canthead, walk, face, stand, watch, die, dead, black, sleep, blood, door, wake, line, wall, kill, water, wind, room, white, sit, hide, grow,bed, fear, lay, rise, hell, sea, meet, scream, pull, death, cut, window, begin, pass, fill, wear, skin, full
Table 3 Representative words of each topic for Japanese lyrics in MCM@I = 50, J = 7. The topic
label indicates our arbitrary interpretation of the representative words. English words are translated by
the authors and original Japanese words are given in parentheses.

z
1

Label
English

2

Scene

3

Exciting

4

Love

5

Clean

6

Lyrical7

Life

Representative words in each topic: top 40 words from P(w|φ s )go, get, let, know, say, night, baby, time, good, way, feel, heart, take, day, dance, make, life, need, party, come, see, tell, dream,
everybody, rock, stop, keep, happy, have, give, tonight, please, world, mind, hand, shake, rain, jump, try, yourtown (machi), night (yoru), rain (ame), summer (natsu), come (kuru), window (mado), white (shiroi), snow (yuki), wait (matsu), room(heya), morning (asa), get back (kaeru), season (kisetsu), fall (huru), spring (haru), winter (huyu), blow (huku), wave (nami), cold
(tumetai), hair (kami), shoulder (kata), memory (omoide), back (senaka), run (hashiru), long (nagai), last (saigo), shadow (kage), sleep(nemuru), close (tojiru), finger (yubi), get wet (nureru), remember (omoidasu), quiet (shizuka), pass (sugiru), cheek (ho), fall (otiru),
breath (iki), open (akeru), car (kuruma)go (iku), front (mae), no, sound (oto), dance (odoru), nothing (nai), fly (tobu), life ( jinsei), can run (hashireru), begin (hajimaru),
proceed (susumu), stand up (tatu), raise (ageru), freedom ( jiyu), era ( jidai), serious (maji), head (atama), body (karada), ahead (saki),power (chikara), throw (suteru), fire (hi), carry (motu), high (hai), take out (dasu), decide (kimeru), ride (noru), speed up (tobasu),
Venus, Japan (nihon), maximum (saikou), rhythm (rizumu), non, up, rise (agaru), party (patatii), wall (kabe), companion (nakama), girl(gaaru), battle (shobu)
love, love (ai), hug (dakishimeru, daku), kiss, feel (kanjiru), girl, pupil (hitomi), ardent (atsui), look on (mitsumeru), sweet, hold, lonely,
sweet (amai), kiss (kisu), pair ( futari), smile, stop (tomeru), miss, sorrowful (setsunai), moon, stop (tomaru), heart (haato), detach(hanasu), overflow (afureru), moment (shunkan), tempestuous (hagesii), moonlight, shine, lovin, touch ( fureru), little, arm (ude), break(kowareru), angel (tenshi), beating (kodo), mystery ( fushgi), destiny, miracle (kiseki), shininsky (sora), dream (yume), wind (kaze), light (hikari), flower (hana), star (hoshi), disappear (kieru), world (sekai), sea (umi), future(mirai), far (toi), voice (koe), moon (tsuki), shine (kagayaku), bloom (saku), flow (nagareru), sun (taiyo), place (basho), blue (aoi),reach (todoku), dark (yami), illuminate (terasu), cloud (kumo), destiny (eien), unstable (yureru), wing (tsubasa), deep ( fukai), song(uta), continue (tuduku), sing (utau), pass over (koeru), shine (hikaru), look up (miageru), bird (tori), finish (owaru), color (iro),
distance (toku), high (takai), rainbow (niji), be born (umareru)
now (ima), mind (kokoro), human (hito), heart (mune), believe (shinjiru), word (kotoba), oneself ( jibun), live (ikiru), tear (namida),
forget (wasureru), love (aisuru), know (siru), hand (te), cry (naku), tomorrow (ashita), walk (aruku), change (kawaru), strong (tsuyoi),
feeling (kimochi), someday (itsuka), kind (yasasii), everything (subete), look (mieru), understand (wakaru), can be (nareru), smile(egao), happy (siawase), can do (dekiru), every day (hibi), outside (soba), crucial (taisetsu), road (michi), eye (me), look for (sagasu),
convey (tutaeru), time ( jikan), take leave (hanareru), guard (mamoru), be able to say (ieru)
good (yoi), say (iu), like (suki), love (koi, daisuki), woman (onna), look (miru), man (otoko), laugh (warau), do (yaru), today (kyo), think(omou), spirit (ki), face (kao), no good (dame), listen (kiku), phone (denwa), tonight (konya), friend (tomodachi), reach (tuku), daughter(musume), bad (warui), meet (au), go (iku), appear (deru), adult (otona), together (issyo), good (umai), consider (kangaeru), die (sinu),
stop (yameru), everyday (mainichi), story (hanashi), talk (hanasu), cheerful (genki), drink (nomu), human (ningen), job (shigoto), early(hayai)the proposed MCM, we examine word probabilities P(w|φz )and topic transition probability P(s|θy,z ) and then visualize topic transition diagrams. To clarify our topic transition
analysis, we manually assigned labels to each topic by ob-

serving the word list whose generative probability P(w|φz )is a large value. Tables 2 and 3 show the assigned labels and representative words for the topics in the English
and Japanese models, respectively. For each topic, we list

WATANABE et al.: MODELING STORYLINES IN LYRICS

1175

Fig. 8 Examples of English MCM (I = 70, J = 11) transitions between topics for each theme (seeTable 2 for word lists). Theme labels are our arbitrary interpretation of their topics and topic transitions.

Fig. 9 Examples of Japanese MCM (I = 50, J = 7) transitions between topics for each theme (seeTable 3 for word lists). Theme labels are our arbitrary interpretation of their topics and topic transitions.

the top 40 words in decreasing order of word probability
P(w|φz ). Figures 8 and 9 show the transition diagrams forsome themes in the English and Japanese models, respectively. Here, each arrow indicates higher transition probabilities (P(s|θy,z ) > 0.20), and each square node indicates
the topic z. Note that the initial node START indicates theinitial state z = 0.
We found the following reasonable storylines with theEnglish model (Fig. 8).
• In theme y = 10, we see the transition Life →Interlude → Feeling.
The topic Interlude
comprises words such as oh, la, and yeah and acts as abridge between the verse and the chorus.
• In theme y = 24, we see that the  Explicit  topic
tends to shift to Exciting, which contains words suchas dance, sexy, and pop. This topic sequence appears
frequently in hip hop/rap songs.
• In theme y = 32, we see the transition Life →Feeling → Love. We arbitrarily decided the theme
label of this topic transition as “Sweet Love”. Here,the last topic Love tends to shift to the first topic
Life. This indicates that the model captures the repetition structure (e.g., A-B-C-A-B-C, where each letter
represents a segment).

We also found the following reasonable storylines with theJapanese model (Fig. 9).
• In theme y = 6, we observe the transition Scene →Lyrical → Love, which is common in love songs.
• In theme y = 12, we see a transition among Life,English, and Exciting, which often appears inJapanese hip hop/rap songs.
• In theme y = 14, we see a transition between Cleanand Lyrical, which is commonly seen in hopefulsongs.
Although we selected these arbitrary diagrams to representa reasonable storyline, in fact, the self-transition diagrams
were trained using other themes. Note that the MCM learns
diﬀerent topic transition distributions according to diﬀerent
themes in an unsupervised manner. This shows that many
lyricists consider the topic order and theme as described in
textbooks [1], [2].
5.

Conclusion and Future Work

This paper has addressed the issue of modeling the discourse
nature of lyrics and presented the first study aiming at capturing the two common discourse-related notions: storylines
and themes. We assumed that a storyline is a chain of transi-

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 20181176

tions over topics of segments and a song has at least one entire theme. We then hypothesized that transitions over topics of lyric segments can be captured by a probabilistic topic
model which incorporates a distribution over transitions oflatent topics and that such a distribution of topic transitions
is aﬀected by the theme of lyrics.
Aiming to test those hypotheses, this study conducted
experiments on the word prediction and segment order prediction tasks exploiting a large-scale corpus of popular music lyrics for both English and Japanese. The findings we
gained from these experiments can be summarized into two
respects. First, the models with topic transitions significantly outperformed the model without topic transitions inword prediction. This result indicates that typical storylines
included in our lyrics datasets were eﬀectively captured as aprobabilistic distribution of transitions over latent topics ofsegments. Second, the model incorporating a latent theme
variable on top of topic transitions outperformed the models
without such variables in both word prediction and segment
order prediction. From this result, we can conclude that considering the notion of theme does contribute to the modelingof storylines of lyrics.
This study has also shaped several future directions.
First, we believe that our model can be naturally extended by incorporate more linguistically rich features suchas tense/aspect, semantic classes of content words, sentiment polarity, etc. Second, it is also an intriguing direction to adopt recently developed word/phrase embeddings [34], [35] to capture the semantics of lyrical phrases ina further sophisticated manner. Third, verse-bridge-chorus
structure of a song is also worth exploring. Our error analysis revealed that the human annotators seemed to be able toidentify verse-bridge-chorus structures and use them to predict segment orders. Modeling such lyrics-specific global
structure of discourse is an intriguing direction of our future work. Finally, it is also important to direct our attention toward the integration of linguistic discourse structureof lyrics and music structure of audio signals. In this direction, we believe that recent advances in music structure
analysis [36], [37] can be an essential enabler.
Acknowledgements
This study utilized the RWC Music Database (Popular Music). This work was partially supported by a Grant-in-Aid
for JSPS Research Fellow Grant Number JP16J05945, andCREST and ACCEL, JST. The authors would like to thankEnago and Dr. Paul Reisert for the English language review.
References[1] D. Austin, J. Peterik, and C.L. Austin, Songwriting for Dummies,
Wileys, 2010.
[2] T. Ueda, The writing lyrics textbook which is easy to understand (inJapanese), YAMAHA music media corporation, 2010.
[3] H.R.G. Oliveira, F.A. Cardoso, and F.C. Pereira, “Tra-la-lyrics: anapproach to generate text based on rhythm,” Proc. 4th International
Joint Workshop on Computational Creativity, pp.47–55, 2007.

[4] A.R. A and S.L. Devi, “An alternate approach towards meaningful
lyric generation in tamil,” Proc. NAACL HLT 2010 Second Workshop on Computational Approaches to Linguistic Creativity, pp.31–
39, 2010.
[5] G. Barbieri, F. Pachet, P. Roy, and M.D. Esposti, “Markov constraints for generating lyrics with style,” Proc. 20th European Conference on Artificial Intelligence (ECAI 2012), pp.115–120, 2012.
[6] C. Abe and A. Ito, “A Japanese lyrics writing support system foramateur songwriters,” Proc. Asia-Pacific Signal & Information Processing Association Annual Summit and Conference 2012 (APSIPAASC 2012), PS.2 - SLA.4 Audio & Music Processing (I), 2012.
[7] D. Wu, K. Addanki, M. Saers, and M. Beloucif, “Learning tofreestyle: Hip hop challenge-response induction via transduction
rule segmentation,” Proc. 2010 Conference on Empirical Methods in
Natural Language Processing (EMNLP 2013), pp.102–112, 2013.
[8] P. Potash, A. Romanov, and A. Rumshisky, “Ghostwriter: Usingan lstm for automatic rap lyric generation,” Proc. 2015 Conferenceon Empirical Methods in Natural Language Processing (EMNLP
2015), pp.1919–1924, 2015.
[9] M. Ghazvininejad, X. Shi, Y. Choi, and K. Knight, “Generating topical poetry,” Proc. 2016 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2016), pp.1183–1191, 2016.
[10] K. Watanabe, Y. Matsubayashi, K. Inui, T. Nakano, S. Fukayama,and M. Goto, “Lyrisys: An interactive support system for writing
lyrics based on topic transition,” Proc. 22nd International Conference on Intelligent User Interfaces (IUI 2017), pp.559–563, 2017.
[11] R. Mayer, R. Neumayer, and A. Rauber, “Rhyme and style features
for musical genre classification by song lyrics,” Proc. 9th International Society for Music Information Retrieval Conference (ISMIR
2008), pp.337–342, 2008.
[12] E. Nichols, D. Morris, S. Basu, and C. Raphael, “Relationships
between lyrics and melody in popular music,” Proc. International
Society for Music Information Retrieval Conference (ISMIR 2009),
pp.471–476, 2009.
[13] E. Greene, T. Bodrumlu, and K. Knight, “Automatic analysis ofrhythmic poetry with applications to generation and translation,”
Proc. 2010 Conference on Empirical Methods in Natural Language
Processing (EMNLP 2010), pp.524–533, 2010.
[14] S. Reddy and K. Knight, “Unsupervised discovery of rhyme
schemes,” Proc. 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2011), pp.77–82, 2011.
[15] K. Watanabe, Y. Matsubayashi, N. Orita, N. Okazaki, K. Inui, S.
Fukayama, T. Nakano, J. Smith, and M. Goto, “Modeling discourse segments in lyrics using repeated patterns,” Proc. 26th International Conference on Computational Linguistics (COLING 2016),
pp.1959–1969, 2016.
[16] J.P.G. Mahedero, Á. MartÍnez, P. Cano, M. Koppenberger, and F.
Gouyon, “Natural language processing of lyrics,” Proc. 13th annual
ACM international conference on Multimedia, pp.475–478, 2005.
[17] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music database: Popular, classical and jazz music databases,” Proc.
3rd of International Society for Music Information Retrieval (ISMIR
2002), pp.287–288, 2002.
[18] B.J. Grosz and C.L. Sidner, “Attention, intentions, and the structureof discourse,” Computational linguistics, vol.12, no.3, pp.175–204,
1986.
[19] D.M. Blei, A.Y. Ng, and M.I. Jordan, “Latent dirichlet allocation,”
Journal of Machine Learning Research, vol.3, pp.993–1022, 2003.
[20] F. Kleedorfer, P. Knees, and T. Pohle, “Oh oh oh whoah! towards
automatic topic detection in song lyrics,” Proc. 9th International Society for Music Information Retrieval Conference (ISMIR 2008),
pp.287–292, 2008.
[21] W. Xu, X. Liu, and Y. Gong, “Document clustering based on
non-negative matrix factorization,” Proc. 26th annual international
ACM SIGIR conference on Research and development in informaion retrieval, pp.267–273, 2003.
[22] S. Sasaki, K. Yoshii, T. Nakano, M. Goto, and S. Morisihima,

WATANABE et al.: MODELING STORYLINES IN LYRICS

1177“Lyricsradar: A lyrics retrieval system based on latent topics oflyrics,” Proc. 15th International Society for Music Information Retrieval Conference (ISMIR 2014), pp.585–590, 2014.
[23] T. Iwata, S. Watanabe, T. Yamada, and N. Ueda, “Topic tracking
model for analyzing consumer purchase behavior,” Proc. 18th International Joint Conference on Artificial Intelligence (IJCAI 2009),
pp.1427–1432, 2009.
[24] D.M. Blei and J.D. Laﬀerty, “Dynamic topic models,” Proc.
23rd International Conference on Machine learning (ICML 2006),
pp.113–120, 2006.
[25] R. Barzilay and L. Lee, “Catching the drift: Probabilistic content
models, with applications to generation and summarization,” Proc.
2004 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies(NAACL HLT 2004), pp.113–120, 2004.
[26] A. Ritter, C. Cherry, and B. Dolan, “Unsupervised modeling of twitter conversations,” Proc. 2010 Conference of the North American
Chapter of the Association for Computational Linguistics - Human
Language Technologies (NAACL HLT 2010), pp.172–180, 2010.
[27] K. Zhai and J.D. Williams, “Discovering latent structure in task-oriented dialogues,” Proc. 52nd Annual Meeting of the Association for
Computational Linguistics (ACL 2014), pp.36–46, 2014.
[28] K. Nigam, A.K. McCallum, S. Thrun, and T. Mitchell, “Text classification from labeled and unlabeled documents using em,” Machinelearning, vol.39, no.2-3, pp.103–134, 2000.
[29] T.P. Minka, “Estimating a dirichlet distribution,” tech. rep., 2000.
[30] S.L. Scott, “Bayesian methods for hidden markov models,” Journalof the American Statistical Association, vol.97, no.457, pp.337–351,
2002.
[31] C.D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S.J. Bethard,and D. McClosky, “The Stanford CoreNLP natural language
processing toolkit,” Association for Computational Linguistics(ACL 2014) System Demonstrations, pp.55–60, 2014.
[32] T. Kudo, K. Yamamoto, and Y. Matsumoto, “Applying conditionalrandom fields to japanese morphological analysis,” Proc. 2004 Conference on Empirical Methods in Natural Language Processing(EMNLP 2004), pp.230–237, 2004.
[33] M. Lapata, “Automatic evaluation of information ordering:
Kendall’s tau,” Computational Linguistics, vol.32, no.4, pp.471–484,
2006.
[34] T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26(NIPS 2013), pp.3111–3119, 2013.
[35] J. Pennington, R. Socher, and C.D. Manning, “Glove: Global vectors
for word representation,” Proc. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pp.1532–1543,
2014.
[36] M. Goto, “A chorus section detection method for musical audio signals and its application to a music listening station,” IEEE Trans. Audio, Speech, Language Process., vol.14, no.5, pp.1783–1794, 2006.
[37] J. Paulus, M. Müller, and A. Klapuri, “Audio-based music structure analysis,” Proc. International Society for Music Information Retrieval Conference (ISMIR 2010), pp.625–636, 2010.

Table A· 1
Notation
Γ(·)
,ζ
w
y¬m
V
Mi,¬m
Nm
Ni,¬m
Nm,v
Ni,v,¬m

Table A· 2
Notation
1(·)
α, β
w
z¬(m,s)J
Szm,s−1 → j,¬(m,s)
Szm,s−1 →∗,¬(m,s)
N(m,s)
N j,¬(m,s)
N(m,s),v
N j,v,¬(m,s)

Table A· 3
NotationVη, ζ, βwx¬(m,s,n)Nm,¬(m,s,n)
Nm,k,¬(m,s,n)

Nym ,¬(m,s,n)

Nym ,wm,s,n ,¬(m,s,n)Nzm,s ,¬(m,s,n)Equation for MUM-CM Inference

The update equations in Algorithm 1 can be rewritten as(9), (10) and (11). Table A· 1 shows the notations in (9)for collapsed Gibbs sampling of theme y in the MUMCM inference. Table A· 2 shows the notations in (10) for
collapsed Gibbs sampling of topic z in the MUM-CM inference. Table A· 3 shows the notations in (11) for collapsed Gibbs sampling of binary variable x in the MUM-CM

Notations in (10) for MUM-CM

Definition
Indicator function
Hyperparameter
Word set in training corpus
Topic set except the s-th segment in the m-th
lyric
# of topics# of segments that trans topic zm,s−1 to j except
the s-th segment in the m-th lyric# of segments with topic zm,s−1 except the s-th
segment in the m-th lyric# of words in the s-th segment in the m-th lyric# of words whose topic label is j except the s-th
segment in the m-th lyric# of a word v in the s-th segment in the m-th lyric# of a word v whose topic label is j except thes-th segment in the m-th lyric

Nzm,s ,wm,s,n ,¬(m,s,n)Appendix A:

Notations in (9) for MUM-CM

Definition
Gamma function
Hyperparameter
Word set in training corpus
Theme set except the m-th lyric
Size of the vocabulary# of lyrics with theme label i except the m-th lyric# of words in the m-th lyric# of the word whose theme label is i except the m-th lyric# of a word v in the m-th lyric# of a word v whose theme label is i except the m-th lyric

Notations in (11) for MUM-CM
Definition
Size of the vocabulary
Hyperparameter
Word set in training corpus
Binary variable set except the n-th binary
variable of the s-th segment in the m-th lyric# of words in the m-th lyric except the n-th
word of the s-th segment in the m-th lyric# of words in the m-th lyric with binary label
k except the n-th word of the s-th segmentin the m-th lyric# of a word whose theme label is ym except
the n-th binary variable of the s-th segmentin the m-th lyric# of a word wm,s,n with theme label ym except the n-th binary variable of the s-th segment in the m-th lyric# of a word whose topic label is zm,s except
the n-th binary variable of the s-th segmentin the m-th lyric# of a word wm,s,n with topic label zm,s except the n-th binary variable of the s-th segment in the m-th lyric

inference.
Appendix B:

Equation for MCM Inference

The update equation in Algorithm 2 can be rewritten as (12).
Table A· 4 shows the notations in (12) for collapsed Gibbs

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.4 APRIL 2018

1178

P(ym = i|y¬m , w, , ζ) ∝ (Mi,¬m + ) · Γ(Ni,v,¬m + Nm,v + ζ)
Γ(Ni,¬m + ζV)·
Γ(Ni,¬m + Nm + ζV) v:N >0
Γ(Ni,v,¬m + ζ)

(9)

m,v

P(zm,s = j|z¬(m,s) , w, α, β) ∝Szm,s−1 → j,¬(m,s) + α
Szm,s−1 →∗,¬(m,s) + αJ·

S j→zm,s+1 ,¬(m,s) + 1(zm,s−1 = j = zm,s+1 ) + α
S j→∗,¬(m,s) + 1(zm,s−1 = j) + αJ

 Γ(N j,v,¬(m,s) + N(m,s),v + β)Γ(N j,¬(m,s) + βV)··
Γ(N j,¬(m,s) + N(m,s) + βV) v:NΓ(N j,v,¬(m,s) + β)
>0

(10)

(m,s),v

Nm,k,¬(m,s,n) + η  Nym ,wm,s,n ,¬(m,s,n) + ζ 1−k  Nzm,s ,wm,s,n ,¬(m,s,n) + β k··Nm,¬(m,s,n) + 2ηNym ,¬(m,s,n) + ζV
Nzm,s ,¬(m,s,n) + βV
Sm 
Sm Γ(SΓ(Si,zm,s →∗,¬m + αJ)
i,zm,s →zm,s ,¬m + Sm,zm,s →zm,s + α)P(ym = i|y¬m , z, α, ) ∝ (Mi,¬m + ) ··Γ(Si,zm,s →∗,¬m + Sm,zm,s →∗ + αJ) Γ(Si,zm,s →zm,s ,¬m + α)s=1P(xm,s,n = k|x¬(m,s,n) , w, η, ζ, β) ∝

(11)
(12)

s =1

Table A· 4
Notation
α,
z
y¬m
J
Mi,¬m
Sm,z→∗
Sy,z→∗,¬m
Sm,z→zSy,z→z ,¬m,

Notations in (12) for MCM

Definition
Hyperparameter
Topic set in training corpus
Theme set except the m-th lyric# of topics# of lyrics with theme label i except the m-th lyric# of segments with topic z in the m-th lyric# of segments whose topic is z and theme is y except
the m-th lyric# of segments whose topic transitions z to z in the m-th
lyric
# of segments whose theme is y and topic transitions z
to z in the m-th lyric except the m-th lyric

sampling of theme y in the MCM inference.

Kento Watanabe
received the B.E. degreein 2013, and M.S. degree in 2015, both fromTohoku University. He is a graduate student
in the Department of System Information Sciences, Tohoku University. His research interests
include machine learning, natural language processing, and human computer iteration.

Yuichiroh Matsubayashi
received his Ph.D
degree in Information Science and Technology
from the University of Tokyo in 2010. He iscurrently a research associate at Tohoku University. His research interests include semantic processing of natural language text. He
has received several awards including the IPSJ
Yamashita SIG Research Award from the Information Processing Society of Japan (IPSJ) andthe Best Journal Paper Awards from the Association for Natural Language Processing (ANLP).
He is a member of the Association of Computational Linguistics (ACL),
ANLP, IPSJ and the Japan Society of Artificial Intelligence (JSAI).

Kentaro Inuireceived his doctorate degreeof engineering from Tokyo Institute of Technology in 1995. He has experience as an assistant professor at Tokyo Institute of Technology and an associate professor at Kyushu Institute of Technology and Nara Institute of Science and Technology, he has been a professorof Graduate School of Information Sciences at
Tohoku University since 2010. His research interests include natural language understanding
and knowledge processing. He currently serves
as the IPSJ director and ANLP director.

Satoru Fukayama
received his Bachelor
degree in Earth and Planetary Physics in 2008,
and Ph.D. degree in Information Science andTechnology in 2013, both from the Universityof Tokyo. He is currently a Senior Researcher
at National Institute of Advanced Industrial Science and Technology (AIST), Japan. His maininterests are in applications of music information retrieval in automated music generations
and choreography synthesis. His approach to research problems is based on machine learningand computational musicology. He has been awarded the Yamashita SIG
Research Award from the Information Processing Society of Japan.

Tomoyasu Nakano
received the Ph.D. degree in Informatics from University of Tsukuba,
Tsukuba, Japan in 2008. He is currently working as a Senior Researcher at the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan. His researchinterests include singing information processing, human-computer interaction, and music information retrieval. He has received several
awards including the IPSJ Yamashita SIG Research Award from the Information Processing
Society of Japan (IPSJ) and the Best Paper Award from the Sound andMusic Computing Conference 2013. He is a member of the IPSJ and the
Acoustical Society of Japan.

WATANABE et al.: MODELING STORYLINES IN LYRICS

1179

Masataka Goto
received the Doctor ofEngineering degree from Waseda University in
1998. He is currently a Prime Senior Researcher
at the National Institute of Advanced Industrial
Science and Technology (AIST), Japan. Overthe past 25 years, he has published more than
250 papers in refereed journals and international
conferences and has received 46 awards, including several best paper awards, best presentation
awards, the Tenth Japan Academy Medal, andthe Tenth JSPS PRIZE.