PACLIC 28

Modeling Structural Topic Transitions for Automatic Lyrics Generation
Kento Watanabe1 , Yuichiroh Matsubayashi1 , Kentaro Inui1 , and Masataka Goto2
1

2

Graduate School of Information Sciences Tohoku University, JAPAN
National Institute of Advanced Industrial Science and Technology (AIST), JAPAN1{kento.w，y-matsu，inui}@ecei.tohoku.ac.jp
2m.goto@aist.go.jp

Abstract

creativity and thereby have promoted automaticsong generation. To date, many individual andgroup musical amateurs have created songs andcommercial activities. To satisfy the demand forcomposer-supportive automatic composition systems and services, various systems, including Orpheus (Fukayama et al., 2012), have been developed.
Furthermore, as musical composition becomes easier, there is a growing need for automatic lyrics generation.

By adopting recent advances in music creation
technologies, such as digital audio workstations and singing voice synthesizers, people
can now create songs in their personal computers. Computers can also assist in creating lyrics or generating them automatically,
although this aspect has been less thoroughly
researched and is limited to rhyme and meter.
This study focuses on the structural relationsin Japanese lyrics. We present novel generation models that capture the topic transitions
between units peculiar to the lyrics, such asverse/chorus and line. These transitions aremodeled by a Hidden Markov Model (HMM)
for representing topics and topic transitions.

However, lyrics generation has yet to be thoroughly explored in the natural language processing
field. While several works have tackled lyrics generation based on lyric-specific characteristics, current
methods are limited to local contexts, such as single
sentences, which cannot capture the overall structure of the generated lyrics (Barbieri et al., 2012; Ramakrishnan A et al., 2009; Reddy and Knight, 2011;
Wu et al., 2013; Greene et al., 2010).

To verify that our models generate contextsuitable lyrics, we evaluate the models using a log probability of lyrics generationand fill-in-the-blanks-type test. The results
show that the language model is far more
effective than HMM-based models, but the
HMM-based approach successfully captures
the inter-verse/chorus and inter-line relations.
In the result of experimental evaluation, ourapproach captures the inter-verse/chorus andinter-line relations.

1 Introduction
Recent music creation technologies such as digital audio workstations and singing voice synthesizers (Kenmochi and Oshita, 2007) have become
immensely popular among enthusiasts of automatically created or vocally synthesized music. These
technologies assist individuals with their musicalThe contribution of our study is twofold: (1) To
more comprehensively understand lyrics generation,
we examine the characteristics or rules by whichpeople identify Japanese lyrics writing and survey
some previous methods. (2) Based on the survey, weconstruct three generation models as an initial step
toward our aim. We focus on two types of information that are essential for lyrics creation: a language
model for lyrics and topic transitions for passages.
Experiments revealed that the language model is
far more effective than models capturing topic transitions. However, by capturing the topic transitions,
we achieve consistency among the topics.

Copyright 2014 by Kento Watanabe, Yuichiroh Matsubayashi, Kentaro Inui, and Masataka Goto
28th Pacific Asia Conference on Language, Information and Computation pages 422–431!422

PACLIC 28

2 Related Work
Previous studies have attempted to reproduce characteristics specific to song lyrics, such as syntax,
rhythm, rhyme, and the relation between melody andtext. Barbieri et al. (2012) adopted a Markov process
to create lyrics satisfying the structural constraintsof rhyme and meter. They also ensured syntacticalcorrectness by a part-of-speech template and computed the semantic relatedness between a target concept and the generated verse/chorus by a Wikipedia
link-based measure. Our model extends Barbieri etal.’s approach to capture not only the semantic relatedness but also the verse-chorus transitions.
Ramakrishnan A et al. (2009) generated melodic
lyrics in a phonetic language (in their case, Tamil).
First, they labeled an input melody with appropriate syllable categories using conditional random
fields and then filled the syllable pattern with words.
Reddy and Knight (2011) developed a languageindependent model based on a Markov process thatfinds the rhyme schemes in poetry and the model
stanza dependency within a poem. However, rhymetransition in their model is used to generate a stanza;
the overall flow of the poem is not captured.
Some researchers have generated lyrics using statistical machine translation. Wu et al. (2013) applied stochastic transduction grammar induction algorithms to generate a fluent rhyming response to
the hip hop challenges allowing various patterns ofmeter. Using a finite-state transducer, Greene etal. (2010) assigned a syllable-stress pattern to every word in each line, subject to metrical constraints.
Moreover, they generated English love poetry andtranslated Italian poetry into English following auser-defined rhythmic scheme.
Although these works capture lyric-specific characteristics to some extent, the structural relations arelimited to lines or local word contexts. To the bestof our knowledge, no existing method accounts forthe semantic relations among large structures, suchas verses and choruses.
Inter-text structural relations are frequently considered in text summarization and conversation
modeling. The summarization technique of Barzilay
and Lee (2004) captures topic transitions in the text
span by a hidden Markov model (HMM), referred toas a content model. Using HMM and a large amount!423

of tweet data, Ritter et al. (2010) and Higashinaka
et al. (2011) modeled the transition of speech acts inan unsupervised manner.

3 Survey on Lyric Writing Techniques
To create a comprehensive model for lyrics generation, we first investigated the characteristics or rules
by which people proceed with lyrics writing in general. We surveyed five textbooks on Japanese lyrics
writing (Endo, 2005; Takada, 2007; Aku, 2009;
Ueda, 2010; Taguchi, 2012) and identified the common features as follows.
3.1 Consistency of Entire Lyrics
The lyrics preferably follow a consistent theme. Authors usually desire to convey a message in theirlyrics, and they reflect their theme in their lyric topics. Frequently, the theme is indirectly expressed
through a concrete story composed of who, what,
when, where, and why information. Each lyric
should be consistent in writing style, such as thepoint of view (first or third person), gender, and date.
3.2 Lyrics and Melody
Lyrics and melody are mutually dependent and influence each other during the creation process. Which
comes first depends on the situation. If developing the melody first, the writer must concentrate on
achieving a suitable melody through rhythm, phonetic length, and lyrical structure. They should also
match the word intonation and accents to the melody
to ensure that their lyrics can be both sung and heard.
Most songs contain some common melodies.
However, listeners may experience dissonancewhen simultaneously hearing upbeat and downbeat
melodies. Thus, the writer needs to share the tone
and atmosphere of his/her lyrics in the same melody.
3.3 Musical Structure of the Lyrics
The structural units of lyrics are verse, bridge, andchorus. Each unit repeatedly appears and shares thesame musical phrases. Consequently, rhythm andmeter are common to shared among the same typeof units. In addition, same-type units are often created as semantically similar topics, such as scene
and emotion, or contrastive topics, such as different
seasons and feelings.

PACLIC 28

In general, each unit plays a typical role in the		storyline. For example, verses often describe a concrete scene or complementary topic that emphasizes
a message in the following chorus. Furthermore, thelines of a single verse/chorus may relay a suitableorder of topic transitions. In the example as follows,
the first and second lines collectively describe a concrete scene. This description is followed by the pro- Figure 1: Example of a mora composed of musitagonist’s reaction to the scene in the third and fourth cal notes. Japanese writers usually compose lyrics
lines.
in such a manner that they can be easily sung by✓✏
Example of relations between lines
the singer. For example, the melody sequence “AA-B” corresponds to “sa-ku-ra” (meaning “cherry(On the way home, it began to snow.)
blossom”).
帰り道 降り始めた雪 ——————— Scene(It is touching your shoulder and melting.)

あなたの肩に 触れて 溶けてゆく—— Scene

the lyrics-melody relation, into their lyrics, some ofwhich are decided in advance. These predetermined
今日もまた あっという間だね —— Sentimentfeatures provide natural inputs to a lyrics generation(The weekend with you is almost over.)task.
あなたとの週末 終わってしまうの – Sentiment
Some previously defined lyrics generation tasks
account of the structural features and lyric-melody
Excerpt from “Everlasting” by Mayo Okamoto✒✑
relation by inputting rhyme and meter. In our approach, the melody is replaced by the mora length
3.4 Balance of Contentsof each phrase. In phonology, a mora specifies the
Emotion and scene are often combined in a period of a sound unit. For example, in Japanese,
verse/chorus. For instance, if a verse/chorus ex- the mora length of the phrase “帰り道, (ka-e-ri-mipresses emotions alone, such as “I love you” and “I chi)” (On the way home) is 5, whereas that of “降want you”, the lyrics are insufficiently balanced to り始めた雪, (fu-ri-ha-ji-me-ta-yu-ki)” (it began toconvey the theme. To ensure that their lyrics are eas- snow) is 8. In a Japanese song, a mora often correily understood and arouse empathy in listeners, writ- sponds to a musical note as shown in Figure 1.
ers should adopt lead-in scenes such as “The roadIn summary, if the input is provided as M line =has been long” and “Reflections in a pool”. Sim- [M0phrase , M1phrase ] = [5, 8], the task generates
ilarly, maintaining the balance between subjective lyrics such as “帰り道 降り始めた雪” (On the way
and objective, concrete and abstract, positive and home, it began to snow).
negative, and universal and novel will prevent egoNow, consider that the input includes partiallycentricity in the lyrics.
composed lyrics. In this scenario, the system partially supports lyrical writer. For example, if the
3.5 Figure of Speech
writer has completed a verse but is unsure of the
Lyrical content is frequently emphasized by figures chorus, the system can generate a chorus that is conof speech such as rhyme, metaphor, double mean- sistent with the completed verse. The experiments
ing, double negatives, interrogatives, onomatopoeia, reported in Section 6 confirm that our models corinversion, repetition, and rewording. The gram- rectly capture the suitable topic transitions.
matical patterns of the lyrical sentence construction
Our lyrics generation task is formally depicted inmarkedly differ from those in the general text.
Figure 2. The task accepts the inputs as follows: (1)previously written parts of the target lyrics including
4 Lyrics Generation Task
an unwritten line and (2) sequences of mora length
As noted in the previous section, songwriters incor- M line = [M0phrase , M1phrase , ...], each correspondporate various features, such as theme, structure, and ing to the mora length of a line to be generated. The(Time flies. Today went by fast, too.)!424

PACLIC 28

Input&(1):&Incomplete&Lyrics&(a) employs a tri-gram model, the models (b) and (c)employ a bi-gram model to avoid data-sparsity due
to the additional conditional parameter, the hidden
state. We explain the details of each model in thenext section.

Input&(2):&Mora!
!

UnwriEen!Line!

(It!is!touching!your!shoulder!and!mel4ng)!
!
!

(Time!ﬂies.!Today!went!by!fast,!too.)!
!

The!mora!length!of!a!line!=![5,8]!

(The!weekend!with!you!is!almost!over.)!
!

5.1 Lyrics Generation Model
Lyrics&Genera9on&Model
Output:&Complete&Lyrics&(On!the!way!home,!it!began!to!snow)!
	!
!

(It!is!touching!your!shoulder!and!mel4ng)!
!
!

(Time!ﬂies.!Today!went!by!fast,!too.)!
!
!

(The!weekend!with!you!is!almost!over.)!
!

Figure 2: Lyrics Generation Task.
output of our lyrics generation task is a line that satisfies the restriction of the input mora.

5 Proposed Method
This section introduces the three generation models
that capture some of the features introduced in previous sections.
In the models, (1) we utilize an n-gram language
model assuming that lyrics are characterized by fluent, easily sung word orderings. In our models,
the n-gram model is conditioned by the appropriate
mora length.
Also, (2) we use a state-transition model assuming that the line and verse/chorus are generated from a consistent, context-dependent word
set. Recall from Subsection 3.3 that each line andverse/chorus are often created as semantically related topics, and that topic transitions between lines
and verses/choruses follow an appropriate order. We
expand the content model (Barzilay and Lee, 2004)
which originally estimates the topic transitions in
documents using a hidden Markov model by assuming that each sentence has its hidden state representing its topic, to capture the inter-verse/chorus andinter-line relations.
Using these two components, we create threemodels illustrated in Figure 3: Although the model!425

The inputs of the lyrics generation model aredemonstrated in Figure 4. The positions of theverse/chorus, line, phrase, and word that should be
generated are defined by i, j, k, and l, respectively.
The mora lengths of the line that should be generline . In Figure 4,
ated is assigned into the variable Mi,jthe second line in the second verse/chorus should be
generated, and the mora length of this line is given
as input. We assigned Linei and Verse to the previously written lines and verses/choruses of the target
lyrics including an unwritten line. These inputs wereapplied to the three generation models as shown in
Figure 3.
(a) The first proposed model is the tri-gram language model P (wl |wl−1 , wl−2 , ml ) with mora restrictions (Equation 1), which assumes that a wordis generated from its predecessors to satisfy the condition of fluent, easily sung lyrics. Note that mlin this model is the mora length of the word andnot the phrase; therefore, the model output is a sequence of the mora word lengths. For example, if
phrasethe input is a mora length of the phrase Mi,j,k=
7, the model should first generate a sequence of themora word lengths [m0 , m1 , m2 , m3 ] = [3, 1, 2, 1],followed by a word sequence [w0 , w1 , w2 , w3 ] = [“あなた (a-na-ta)”, “の (no)”, “肩 (ka-ta)”, “に (ni)”(your shoulder)]. Therefore, we specified that asequence of words with the mora length m is genphrase
erated with some probability P (m|Mi,j,k). Thus,
we have m = [m0 , ..., ml , ...].
line
P (Linei,j |Mi,j
)=

line
|Mi,j
|

!

k=0

phraseP (m|Mi,j,k
)

|m|
!
l=0

P (wl |wl−1 , wl−2 , ml )(1)(b), (c) The second and third proposed models
is implemented for generating a consistent lyric.
In generating a consistent lyric as described in
Subsection 3.3, the topic transitions between lines
and verses/choruses must be estimated. In this

PACLIC 28(a)&Tri?gram&Model&with&the&mora&restricAon
Input&1:the&sequence&of&the&mora&length&of&phrasephraseM

i,j,0

phraseM

i,j,1

…(c)&Bi?gram&+&Verse&Content&Model&

Input&1:the&sequence&of&the&mora&length&of&phrase

M …

phrasephraseM

i,j,2

The&sequence&of&mora&length&of&word

… m(b)&Bi?gram&+&Line&Content&Model&

i,j,0

Input&1:the&sequence&of&the&mora&length&of&phrase

M …

M

ml#1

ml

wl#2

wl#1

wl

…

… m

…

…

…

… m

…

…

ml#1

ml

wl#2

wl#1

wl

line

Ci,+j+1 …

line
i,+j#1line

Ci,+j

The&sequence&of&hidden&state… Linei,j#1

i,j,0

phraseM

i,j,1

phraseM …
i,j,2

The&sequence&of&mora&length&of&word

l#2

… C

M

i,j,2

i,j,1

The&sequence&of&mora&length&of&word

l#2

phrasephrasephrase

l#2

ml#1

ml

wl#2

wl#1

wl

verse

Ci+1

… C

verse
i#1

Ci

verse

………

The&sequence&of&hidden&state

Line
i,j

… Verse Verse Verse …

…
Line
i,j+1

Input&2:&previously&wri2en&line&including&an&unwri2en&line

i#1

i

i+1

Input&2:&previously&wri2en&verse&including&an&unwri2en&line

Figure 3: Lyrics generation model: (a) The n-gram language model generates a word given the line’s moraline = [M phrase , M phrase , ...]. The sequence of the mora lengths of the word [m , m , ...] islength Mi,j
0
1
i,j,0
i,j,1
phrasegenerated from Mi,j,k. (b), (c) Based on the content model (Barzilay and Lee, 2004), the generation
model captures the relations between lines and verses/choruses. The transition sequence of the hidden stateline , ..., C line , ...] or [C verse , ..., C verse , ...] is estimated by specifying the context (already composed lines
[Ci,0
0
i,j
i
or already composed verses/choruses) and applying the Viterbi algorithm. Finally, the words are generated
line and C verse , the mora length of the word m , and the previous word w
from each hidden state Ci,j
l
l−1 .
i

study, the hidden state transitions between lines
and verses/choruses in Japanese lyrics were learned
by a content model (Barzilay and Lee, 2004).
The features of the content model were bag-ofword-unigram containing the top 5,000 words in the
training set, determined in a preliminary experiment.
The hyper parameter in the content model training
was set to 0.01. Next, we obtained the sequenceline , ..., C line , ...]
of hidden states Cline= [Ci,0ii,jand Cverse = [C0verse , ..., Civerse , ...]; these
are the topic transitions obtained by theViterbi algorithm given the preferably written parts Linei = [Linei,0 , ..., Linei,j , ...] andVerse = [V erse0 , ..., V ersei , ...] including an
unwritten line. Finally, we specify the word generation probabilities P (wl |wl−1 , Civerse , ml ) andline , m ) to generate a word belonging
P (wl |wl−1 , Ci,jlline (Equations 2
to the hidden state Civerse and Ci,jand 3). In this study, a fluent, easily sung lyric
has been generated from the previous word wl−1 ,the mora length ml of the word, and the hiddenline . In contrast, the algorithms ofstate Civerse or Ci,j!426

Barzilay and Lee (2004), Ritter et al. (2010), andHigashinaka et al. (2011) use only the hidden state.
line
line
P (Linei,j |Ci,j, Mi,j
)=

line
|Mi,j
|

!

phraseP (m|Mi,j,k
)

k=0

|m|
!
l=0

line
P (wl |wl−1 , Ci,j, ml )(2)line
P (Linei,j |Civerse , Mi,j
)=

line
|Mi,j
|

!

k=0

phraseP (m|Mi,j,k
)

|m|
!
l=0

P (wl |wl−1 , Civerse , ml )(3)

Although it appears that this method is restricted tothe hidden state estimation for only one unwrittenline, it is possible to extend this method for multipul
unwritten lines by repeatedly applying the Viterbi algorithm after generating one line.
5.2 Model Estimation
Our generation model is estimated by maximum
likelihood (Equations 4 and 5). The count(*, w)
returns the number of the occurrences of the word

PACLIC 28✓

The position of verse/chorus that should be generated: i = 1✏

The position of line that should be generated: j = 1
The position of phrase that should be generated: k
The position of word that should be generated: lphrasephrasephraseline
The mora length of line: M1,1
= [M1,1,0, M1,1,1, M1,1,2] = [7, 3, 5]

The previously written lines including an unwritten line: Line1 = [Line1,0 , Line1,1 , Line1,2 , Line1,3 ]Line1,0 =“帰り道 降り始めた雪” (On the way home, it began to snow.)
Line1,1 =Unwritten Line
Line1,2 =“今日もまた あっという間だね” (Time flies. Today went by fast, too.)
Line1,3 =“あなたとの週末 終わってしまうの” (The weekend with you is almost over.)The previously written verses/choruses including an unwritten line: Verse = [V erse0 , V erse1 , V erse2 ]V erse0 =“何も言わず そっと 肩を抱き寄せて, ...” (Please hug me softly, ...)
V erse1 = Line1 = [Line1,0 , Line1,1 , Line1,2 , Line1,3 ]V erse2 =“5 分だけ, あと 10 分だけ, ...” (Just 5 minutes, just 10minutes, ...)✒

Excerpt from “Everlasting” by Mayo Okamoto

Figure 4: Example of the model input.

w (or a hidden state), and the Wml is the word set
with the mora length ml . To avoid the word sparseness problem, these probabilities are smoothed byGood-Turing discounting using SRILM (a toolkit
for building and applying statistical language models) (Stolcke, 2002).
PM L (wl |wl−1 , wl−2 , ml ) =count(wl−2 , wl−1 , wl )!
w∈Wm count(wl−2 , wl−1 , w)

(4)

PM L (wl |C, wl−1 , ml ) =count(C, wl−1 , wl )!
w∈Wm count(C, wl−1 , w)

(5)

count(m)phrase count(m)
m∈M

(6)

l

l

The generated sequence of the mora word lengths
is simply estimated by maximum likelihood (Equation 6).
phraseP (m|Mi,j,k
)= !

i,j,k

6 Experiments
6.1 Evaluation Measure
The evaluation measure is another open problem inlyrics generation. Barbieri et al. (2012) and Wu et

!427

✑

al. (2013) evaluated the generated lyrics by humanannotation. However, because manual evaluations
are expensive and time consuming, they are limited
to a small number of test instances. Furthermore, theevaluation measures of an artistic quality strongly
depend on the individuals; therefore, to achieve an
evaluation measure of an adequate quality, copious
annotation is required.
We evaluated our generation model by two different measures: log probability of the original line andfill-in-the-blanks-type testing. In the log-probability
measure, we assumed that among all possible lines,
the original line is generated with the highest probability. To calculate the log probability, the topic transition [C0 , C1 , ...] was predetermined by providing
the lines or verses/choruses and applying" the Viterbi#
algorithm. The log probability, log P (Line) ofgenerating each line was then calculated as the logarithm of Equations 1, 2, and 3.
The fill-in-the-blanks-type test evaluates whether
the correct line, selected from two candidates, is inserted into a given hidden line. One of the candidates is a correct answer randomly selected fromthe original song. The other candidate is an incorrect answer with the same mora length as the linefrom the original song but is randomly selected fromanother song. The candidate scoring the highest

PACLIC 28-61

Log-probability
Accuracy

0.6

-62

-385

0.54

-400

Log-probability-63
AccuracyLog-probability

0.56

0.52

-410

0.5
90 100102030 40 50 60 70
Number of states

80

0.58

0.56

-64

0.55

-65

0.54
0.53

-66

-405

0.59

0.57

0.58
-390
-395

Log-probability
Accuracy

Accuracy

-380

0.52
-67
-68

0.51
10

20

30

40 50 60 70
Number of states

80

0.5
90 100Figure 5: Log probability of whole lyrics generation (black circles) and the accuracy of a fill-in-theblanks-type test evaluated on the development set,with the content model restricted to verses and choruses (gray diamonds).

Figure 6: Log probability of verse/chorus generation (black circles) and the accuracy of a fill-in-theblanks-type test evaluated on the development set,with the content model operated in the line mode(gray diamonds).

log probability is predicted as the correct answer.
This measure checks whether the proposed model
correctly captures topic transitions in each line orverse/chorus.

over the entire lyrics. Similarly,! the log probabil"
ity shown in Figure 6 is the log P (verse/chorus) ,calculated as the sum of the logarithm of Equation 3
in an entire verse or chorus.
In each case, the log probability decreases as
the number of states increases because the bi-gramcounts face data sparsity. However, the accuracyof the fill-in-the-blanks-type test monotonically increases and almost saturates at 10 states. Consequently, we specified 10 states in each content
model.

6.2 Dataset
The experiments were performed on Japanese popular music lyrics covering various genres, suchas Enka 1 and 1970s pop. Because our algorithm has limited capacity for calculating the moralength, foreign language songs were excluded in advance. The dataset contains 24,000 songs, 136,703
verses/choruses, 411,583 lines, and 61,118 words.
We allocated 20,000 songs to the training set andreserved 2,000 songs each for development and testing.
6.3 Number of Hidden States
Prior to evaluation, the numbers of the hidden statesin the two content models needed to be strategicallyselected to optimize the accuracy of the fill-in-theblanks-type test on the development set. Figures 5
and 6 show the average log probabilities and accuracies of the fill-in-the-blanks-type test when applying
this test to each content model. Note
! that the log"
probability shown in Figure 5 is log P (Lyrics) ,calculated as the sum of the logarithm of Equation 2
1

Enka is a genre of a Japanese traditional ballad.

!428

6.4 Evaluation
Table
the average log probability,! 1 lists"
log P (Line) , of line generation in each model,
evaluated on the development and test data.
Although the content models partially include
a bi-gram language model, the tri-gram modelyielded the best performance. This result indicatesthe superior effectiveness of line generation by thelanguage model than by the contents models.
Nonetheless, the content models capture a suitable order of lines. The average accuracy of the fillin-the-blanks-type test is tabulated in Table 2. In thistask, the counter-candidate line is randomly selectedfrom another song and thus almost grammatical inconstruct. The main clue for accurate selection is asemantic relation between the topics. In this situation, the accuracy of the tri-gram model is equiva-

PACLIC 28

Model
Tri-gram Model
Content model for Line
Content model for Verse

Dev-28.28-38.36-33.03Test-29.02-38.89-33.84

Interpretation
Scene
MemorySorrow & Love

Table 1: Log probability of line generation.
Model
Tri-gram Model
Content model for Line
Content model for Verse

Dev
50.80%
56.89%
57.91%

Life & World
Dream & FutureTest
47.97%
56.14%
56.69%

1970s pop

Table 2: Accuracy of the fill-in-the-blanks-type test.

Enka
traditional ballads
Modern Song

Representative words
town (machi), room (heya), city (tokai),sunset (yuuhi), run (hashiru),
remember (wasurenai), memory (omoide),met (deatta), nostalgic (natsukasii),love (koi), express (iu), cry (naku),
affections, sentiment, mind (kimochi),live (ikiru), future (mirai), bravery (yuuki),
destination (yukusaki), reality (genjitsu),
dream (yume), future (mirai),new (atarashii), world (world), one (hitotsu),lie (uso), romance (romansu), rose (bara)lullaby (rarabai), kiss (kuchiduke),
human life (jinsei), harbor (minato),sake (sake), old home (kokyou),paradise (paradaisu), cute (kawaii),
drama (dorama), dance (danse),

Table 4: Representative Words in each Semantic Class.
lent to the chance rate. On the other hand, both content models significantly improve the performanceof lyrics generation.
We also qualitatively analyzed the obtained hidden states and their transitions in the content models. Table 3 illustrates the state transition table in theverse/chorus mode with 10 states. To clarify the discussion, we manually assigned easy-to-understandlabels to the states and representative words to eachhidden state (see Table 4).
Our content model in the verse/chorus mode concurrently learns two types of hidden states (Table 3).
The first type corresponds to specific music genres; the second corresponds to specific tendencies ofword appearances. The model successfully captures
music genres with particular stylistic and vocabulary
characteristics, such as 1970s pop, Enka, and modern songs. Once a current state shifts into one ofthese states, it rarely shifts to another state. This indicates that the model generates suitable words thatconsistently fit the target genre.
Secondly, some states successfully capture thetopics where the transition probabilities between
them have some tendency; state transition probabilities are not random but instead biased against
semantically related contents. As seen in Table 3,
this type embraces five states, namely, Scene, Memory, Sorrow & Love, Dream & Future, and Life &World. In these topics, (1) the self-transition is themost likely one. (2) The transition probability fromSTART to Scene is relatively high, and the transition probability from Scene to END is relatively low
compared with the ones from others to END. (3)!429

The transition probabilities from Memory are almost
even except for the self-transition and the transition
to specific music genres (1970s pop, Enka, and Modern song). Therefore, Memory tends to play the roleof an intermediate content in a lyrics. (4) The transition probability from Sorrow & Love, Life & World,and Dream & Future to END is relatively high. Thus
the last verse/chorus in whole lyrics tends to becomethese three states. (5) Life & World and Dream &Future are strongly correlated. This indicates thatthe words representing hopes and bright futures tend
to appear side by side.

7 Conclusion and Future Works
In this study, we presented content models for automatic lyrics generation that capture topic transitionsin individual lines or verses/choruses. The content
models are less capable of computing original line
probabilities than the tri-gram model but better capture the inter-verse/chorus and inter-line relations.
Currently, each model is separately constructed but
the result suggested that combining these models
would improve topic consistency. A multi-modal
approach combining musical and lyrics informationis also worthy of consideration. Some previous researchers have generated lyrics from musical information (Mihalcea and Strapparava, 2012; Hannah Davis, 2014). Musical information other thanmora (such as rhyme, rhythm, melody, and chord)will be incorporated in the next version of our structured model.

State before transition

PACLIC 28

START
Scene
MemorySorrow & Love
Life & World
Dream & Future
1970s pop
Enka
ModernEND

SceneMemory

9.0%
9.1%
25.1%
14.0%
18.5%
16.1%
28.8%
12.7%

26.6%
37.7%
11.6%
5.1%
5.1%
5.5%
10.5%
0.9%
6.5%

11.3%
12.9%
38.1%
12.5%
6.3%
6.9%
3.6%
0.3%
1.2%

State after transition
Sorrow
Life
Dream& Love & World & Future
2.4%
9.0%
11.5%
3.7%
7.6%
9.7%
12.6%
10.2%
11.0%
32.1%
6.1%
13.2%
4.5%
49.2%
13.5%
7.0%
11.9%
45.5%
4.9%
2.0%
4.5%
0.0%
0.4%
0.8%
0.6%
8.1%
1.2%

1970s pop

Enka

10.3%
10.8%
5.0%
4.6%
1.3%
2.5%
52.0%
1.4%
3.9%

17.4%
1.2%
0.3%
0.1%
0.2%
0.5%
1.3%
65.5%
1.4%

Modern
Song
10.3%
5.0%
1.1%
0.5%
5.0%
0.7%
2.9%
1.3%
62.1%

Table 3: Transition table between the hidden states of a verse/chorus. The vertical axis represents the hidden
states before transition. The horizontal axis represents the hidden states after transition. Each cell contains
the transition probabilities between the hidden states. The top three transition probabilities are shown inbold. To simplify the table, we omit hidden states that are erroneously reached from the start state.

References
Yu Aku. 2009. Sakushi Nyumon (Introduction to Writing
the Lyrics). Iwanami Shoten.
Gabriele Barbieri, Franois Pachet, Pierre Roy, andMirko Degli Esposti. 2012. Markov constraints forgenerating lyrics with style. In Luc De Raedt, Christian Bessire, Didier Dubois, Patrick Doherty, Paolo
Frasconi, Fredrik Heintz, and Peter J. F. Lucas, editors, ECAI, volume 242 of Frontiers in Artificial Intelligence and Applications, pages 115–120. IOS Press.
Regina Barzilay and Lillian Lee. 2004. Catching thedrift: Probabilistic content models, with applications
to generation and summarization. In Proceedings of
HLT-NAACL, pages 113–120.
Kozo Endo. 2005. Sakushi Hon (Bool for Writing the
Lyrics). Shinko-Music Entertainment.
Satoru Fukayama, Daisuke Saito, and Shigeki Sagayama.
2012. Assistance for novice users on creating songs
for japanese lyrics. In in Proceedings of the International Computer Music Association 2012.
Erica Greene, Tugba Bodrumlu, and Kevin Knight. 2010.
Automatic analysis of rhythmic poetry with applications to generation and translation. In Proceedings ofthe 2010 Conference on Empirical Methods in Natural
Language Processing, EMNLP ’10, pages 524–533,
Stroudsburg, PA, USA. Association for Computational
Linguistics.
Saif M. Mohammand Hannah Davis. 2014. Generatingmusic from literature. pages 1–10. Workshop on Computational Linguistics for Literature.
Ryuichiro Higashinaka, Noriaki Kawamae, Kugatsu
Sadamitsu, Yasuhiro Minami, Toyomi Meguro, Kohji Dohsaka, and Hirohito Inagaki. 2011. Building
a conversational model from two-tweets. In Workshop!430

on Automatic Speech Recognition and Understanding,pages 330–335.
Hideki Kenmochi and Hayato Oshita. 2007. Vocaloid― commercial singing synthesizer based on sample
concatenation. In in Proceedings of Interspeech 2007,
pages 4009–4010.
Rada Mihalcea and Carlo Strapparava. 2012. Lyrics, music, and emotions. In Proceedings of the 2012 Joint
Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 590–
599, Stroudsburg, PA, USA. Association for Computational Linguistics.
Ananth Ramakrishnan A, Sankar Kuppan, andSobha Lalitha Devi. 2009. Automatic generation of tamil lyrics for melodies. In Proceedings of theWorkshop on Computational Approaches to Linguistic
Creativity, CALC ’09, pages 40–46, Stroudsburg, PA,
USA. Association for Computational Linguistics.
Sravana Reddy and Kevin Knight. 2011. Unsupervised discovery of rhyme schemes. In ACL (Short Papers)’11, pages 77–82. Association for Computational
Linguistics.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Human
Language Technologies: The 2010 Annual Conferenceof the North American Chapter of the Association for
Computational Linguistics, HLT ’10, pages 172–180,
Stroudsburg, PA, USA. Association for Computational
Linguistics.
Andreas Stolcke. 2002. Srilm - an extensible language
modeling toolkit. pages 901–904.
Shun Taguchi. 2012. Omoidori Ni Sakushi Ga Dekiru
Hon (Book tha You Can Write the Lyrics). RittorMusic.

PACLIC 28

Motoki Takada. 2007. Sakushi No Kotsu Ga Wakaru(You Understand the How to Write the Lyrics). Chuo
Art Publishing.
Tatsuji Ueda. 2010. Yoku Wakaru Sakushi No Kyoukasho(The Textbook of the Lyrics). Ymaha Music Media.
Dekai Wu, Karteek Addanki, Markus Saers, and Meriem
Beloucif. 2013. Learning to freestyle: Hip hop
challenge-response induction via transduction rulesegmentation. In Proceedings of the 2013 Conferenceon Empirical Methods in Natural Language Processing, pages 102–112, Seattle, Washington, USA, October. Association for Computational Linguistics.

!431