A Melody-conditioned Lyrics Language Model
Kento Watanabe1 , Yuichiroh Matsubayashi1 ,Satoru Fukayama2 , Masataka Goto2 , Kentaro Inui1,3 , Tomoyasu Nakano2
1
Graduate School of Information Sciences, Tohoku University,
2
National Institute of Advanced Industrial Science and Technology (AIST),3
RIKEN Center for Advanced Intelligence Project{kento.w, y-matsu, inui}@ecei.tohoku.ac.jp,{s.fukayama, m.goto, t.nakano}@aist.go.jp
Flute,⁄ @@ , - )2 ) ) ) ) ) ) ) ) ) ) ! - ) ) #) ) (!
Example of awkward
lyrics.
,⁄ , - )2 ) ) ) ) ) ) ) ) ) ) , ) ) ) ) (!
ma-da shi- ra
na- i
a- shi・ ta e yu-ku) ) ) (not)まだ) ) 知らへ)ない ) ) (tomorrow)) 明日) ! - ) (to)) #)行く) (!
,)2 ) )(know)⁄ @, , -(yet)
(go)
)
)
)
)
!
)
)
)
)
)
)
)
(!
,
#)
2
)
⁄@
) to)an unknown) (Proceed) ) lyrics.
) , ) ) ) ) (!
) ) ) tomorrow),
Example⁄ ,, -- )2of)) natural,⁄ , - ))2) )) )) )) )!) . ) ) )#) )) )) )) -, ) )) )) )) )) ((!
7
de
i- ta
ko-no mi-chi⁄ , - hi-to-ri
7一人) で ) a-) ru-)(walked)
この)!歩いた) ! )) #))(road))) )) )) ! - 7(this))道 '(!
,)2)) ) )) (FUNC)⁄ , )(alone)
#)
)
)
)
)
)
)
7
⁄ , - ) ) ) )(I walked
alone...
Thisroad)) ) ) ) ) ) , 7 ) ) ) ) (!
⁄ )2! ) ) ) ) #) ) )! ) ) ) ) ! - ) ! ) ) ) )! ., ) ))⁄ , - ) ) ) ) ) )! . ) #) ) ) ) - ) ) ) ) ) (7
Figureand 7natural )lyrics.
7 ! ) of) ) awkward!
⁄ ) ! ) 1:)7 #) Examples
)
#)
#)
)
)
)
)
)
)
)
)!
)
)
)
) - )7) song-) 7) 'isFUNC
a function
word.
The
!
⁄ , ) indicates
)
#)
)
)
)
!
)
)
)
)
)
)
)
- 7Database- )7 ) ! ) ) ) ) ! ) ) )
from
Music
(RWC-MDB-P-2001⁄ ) ! the
)RWC
)
)
)!
)!
)
(
)
7
⁄
No.20) (Goto et al.,) ! 2002).
#) ) )! ) , ) ) ) ! - ) ! ) )7 ) )! . )) )
)
)
)
)
⁄
!#) )
7
⁄ () ! ) ) #), )) )) () !! ) ) ) ) - #))7 ) ))! ) ) )) )) -) )7) ') ! #) ) ) ) - )⁄7languagein lyrics and to design a computer sys⁄
!
)
)
)
!
)
)
)!
)!
!
)
)
)
)
)
)7 (in writing.
) , They) propose 7 )- lyricists
tem⁄ )for assistingto⁄constrain
their( ! models to) generate) )only) ) lyrics) ') conditions, )given- 7 on) !#)syllable⁄ ( satisfy
thatcounts,⁄ positions, etc. However, such constraints
rhyme⁄
are assumed to be manually provided by a humanuser,⁄ which requires the user to interpret a source
melody and transform their interpretation to a setof⁄constraints. To assist users with transforming a1

AbstractFl5This paper presents a novel, data-driven language model that produces entire lyrics for agiven input melody. Previously proposed models for lyrics generation suffer from the inability of capturing the relationship between
lyrics and melody partly due to the unavailability of lyrics-melody aligned data. In
this study, we first propose a new practical method for creating a large collection of
lyrics-melody aligned data and then create acollection of 1,000 lyrics-melody pairs augmented with precise syllable-note alignments
and word/sentence/paragraph boundaries. We
then provide a quantitative analysis of thecorrelation between word/sentence/paragraphboundaries in lyrics and melodies. We thenpropose an RNN-based lyrics language model
conditioned on a featurized melody. Experimental results show that the proposed model
generates fluent lyrics while maintaining thecompatibility between boundaries of lyrics
and melody structures.

1

Rest

Fl

9
Flute

1
Fl

13
Fl

5
Fl

Rest

17
Fl

9
Fl

21
Fl

13
Fl
25
Fl

17
Fl
29
Fl

21
Fl
33
Fl

25
Fl
37
Fl

29
Fl
41
Fl

33
Fl
45
Fl

37
Fl
49
Fl

41

Fl

45

Introduction

Fl

49

Writing lyrics for a given melody is a challenging
task. Unlike prose text, writing lyrics requires bothknowledge and consideration of music-specificproperties such as the structure of melody, rhythms,
etc. (Austin et al., 2010; Ueda, 2010). A simple example is the correlation between word boundaries
in lyrics and the rests in a melody. As shown in
Figure 1, a single word spanning beyond a long
melody rest can sound unnatural. When writing
lyrics, a lyricist must consider such constraints incontent and lexical selection, which can impose
extra cognitive loads.
This consideration when writing lyrics has motivated a wide-range of studies for the task ofcomputer-assisted lyrics writing (Barbieri et al.,2012; Abe and Ito, 2012; Potash et al., 2015; Watanabe et al., 2017). Such studies aim to model the

melody to constraints, a language model that automatically captures the relationship between lyrics
and melody is required.
Some studies (Oliveira et al., 2007; Oliveira,2015; Nichols et al., 2009) have quantitatively analyzed the correlations between melody and phonological aspects of lyrics (e.g., the relationship between a beat and a syllable stress). However, thesestudies do not address the relationship between
melody and the discourse structure of lyrics. Lyrics
are not just a sequence of syllables but a meaningful
sequence of words. Therefore, it is desirable thatthe sentence/paragraph boundaries are determinedbased on both melody rests and context words.
Considering such line/paragraph structure of
lyrics, we present a novel language model that gen-

163
Proceedings of NAACL-HLT 2018, pages 163–172
New Orleans, Louisiana, June 1 - 6, 2018. c 2018 Association for Computational Linguistics

,⁄ @@ , - )2 ) ) ) ) ) ) ) ) ) ) ! - ) ) #) ) (!
,⁄ , - )2 ) ) ) ) ) ) ) ) ) ) , ) ) ) ) (!
,⁄ , - )2 ) ) ) ) ) ) ) ) ) ) ! - ) ) #) ) (!
) ) ) ) , ) ) ) ) (!
,⁄ ,@ ,- -)何2 )2 )) )) か))) ))足り)) ない) ! - )た) #) ) (!
,⁄ @ )) ) ) ) )! ) #)) )) と)) )思っ
)
)
)
(
)
,
)
.
)
⁄ , - 7 2 ) ) ) ) 7) ) ) ) ) ) , 7 ) ) ) ) (!
,⁄ ) )) ) ) ) ) ) ! #) ) ) ) ) ! ) ) ) ',)⁄ , - ) ) ) ) ) )7 ) ) ) ) ! -7 ) ) #) ) (!
⁄ )2 ) ! ) ) ) ) #) ) )! ) ) ) ) ! ) ! ) ) ) , ) )⁄ , - (I thought) something) ) ) )was) missing)) ) )-)) )(!! )!7 .,,⁄ ) ! ) ) )#)2 ) )) )) ) that
!
#)
#) )
⁄ , - ) ) ) ) ) ))!). ) )) )#) ) )) )!) ) -) )) )) ) -) 7) ( ) ) - )7
7 ) ( using
Figure 2: Melody-lyrics
Needle⁄ ) ! ) ) 7) )! )!alignment
7the) ) ! )7 ) ) )
!
)
)
)
)
,
7
!
man Wunsch⁄algorithm.
BOL
denotesalineboundary.
)
#)
)
)
)
!
)
)
⁄ (, ) , ) ) )) ()! - )7 - ) ))!#) - ))7 )) ) ) )) ''⁄! ) ) ) ) #)7 ) )! ) ) ) ) ! - ) ! ) ) ) )! . ) ))⁄(i.e., word/sentence/paragraphboundaries) of a 7⁄) ! #) ) ) ) - )
!
#)
!
#)
)
)
)
)
)
)
) )of)!Figure) ) in) the bottom) ) ) ) -2.)7 We
lyric, as illustrated⁄
7
create such⁄a dataset by automatically combining
!
)
)
)
!
)
)
)!
)!
!
)
)
)
)
)
) available)7 ( online) , forum) sites: 7 )- from
two types of⁄data⁄ (score data
digital music!#) ) ) ) 2)) ) and)7 of )Figure( !(the top
'
)
)
,
⁄
raw lyrics data (the middle).
1

2

Melody-lyric alignment data

Our goal is to create a melody-conditioned language model that captures the correlations between
melody patterns and discourse segments of lyrics.
The data we need for this purpose is a collection of melody-lyrics pairs where the melody andlyrics are aligned at the level of not only notesyllable alignment but also discourse components

Digital musical score data with syllables
5

na ni ka tarina i to9

Rest

Syllable

Rest

Melodyo mo o ta

Lyric text data
NULL
NULL
NULL
with syllable andNeedleman-Wunsch alignment algorithm
NULL
boundary
13[na-ni] [ka] [ta-ri] [na-i] [to] [o-mo] [ta]Syllable1
Word 〈BOL〉
17

Melody-Lyric alignment data
5

Syllable
Word

21

9

na-ni ka ta- ri na- i
何か 足り ないtoと25(some- (FUNC)(enough)thing)(FUNC)

(not)

Rest

MelodyRest〈BOL〉

erates lyrics whose word, sentence, and paragraphboundaries are appropriate for a given melody,
without manually transforming the melody to syllable constraints. This direction of research hasreceived less attention because it requires a large
dataset consisting of aligned pairs of melody andsegment boundaries of lyrics which has yet to exist.
To address this issue, we leverage a publiclyavailable collection of digital music scores and create a dataset of digital music scores each of which
specifics a melody score augmented with syllable
information for each melody note. We collected
1,000 Japanese songs from an online forum where
many amateur music composers upload their musicscores. We then automatically aligned each musicscore with the raw text data of the corresponding
lyrics in order to augment it with the word, sentence, and paragraph boundaries.
The availability of such aligned, parallel data
opens a new area of research where one can conduct a broad range of data-oriented research forinvestigating and modeling correlations between
melodies and discourse structure of lyrics. In thispaper, with our melody-lyrics aligned songs, we investigate the phenomena that (i) words, sentences,and paragraphs rarely span beyond a long melody
rest and (ii) the boundaries of larger components(i.e., paragraphs) tend to coincide more with longer
rests. To the best of our knowledge, there is noprevious work that provides any quantitative analysis of this phenomenon with this size of data (seeSection 7).
Following this analysis, we build a novel, datadriven language model that generates fluent lyrics
whose sentence and paragraph boundaries fit an
input melody. We extend a Recurrent Neural Network Language Model (RNNLM) (Mikolov et al.,2010) so that its output can be conditioned on afeaturized melody. Both our quantitative and qualitative evaluations show that our model captures
the consistency between melody and boundaries oflyrics while maintaining word fluency.

o-mo
思っ

ta
た

(think) (FUNC)

13

29

17

33

21

37

25

41

29

45

33

49

37

A digital ⁄music score specifies a melody scoreaugmented with syllable information for each
melody note⁄ (see the top of Figure 2). Score data
augmented in this way is sufficient for analyzing⁄ between the phonological aspects
the relationshipof lyrics and melody, but it is insufficient for ourgoal since the structural information of the lyrics
is not included. We thus augment score data further with boundaries of sentences, and paragraphs,
where we assume that sentences and paragraphsof lyrics are approximately captured by lines andblocks,1 respectively, of the lyrics in the raw text.
The integration of music scores and raw lyrics
is achieved by (1) applying a morphological analyzer2 to raw lyrics for word segmentation andChinese character pronunciation prediction and (2)aligning music score with raw lyrics at the syllable level as illustrated in Figure 2. For this alignment, we employ the Needleman-Wunsch algorithm (Needleman and Wunsch, 1970). This alignment process is reasonably accurate because it fails
in principle only when the morphological analysis
fails in Chinese character pronunciation prediction,
which occurs for only less than 1% of the words inthe data set.
With this procedure, we obtained 54,181
Japanese raw lyrics and 1,000 digital musical

164

41

45

49

1

Blocks are assumed to be segmented by empty lines.
To extract word boundaries and syllable information forJapanese lyrics, we apply MeCab parser (Kudo et al., 2004).
2) ) - ) #) -) (!) #) ) (! , ,⁄⁄, -⁄)2)2⁄ ,)2 - )2⁄⁄, , -⁄- )2,)2⁄))-)),))2)-)) ))2)) ) )) ) )) ) )) ))) ) )))! ), -)) !)) )-)! )#),)-))))#)(!))))#) ))(! (! , , ,) ) ) #) )#) ) ) ) )(!)) ) (! (!
⁄⁄, , ⁄--)7,))2⁄)- ),))2))- )7)2)!7) . )) ) )!7) .) )) ))) -, ))7 ) ),) -), ))7 ) ( ) ( , , ,3 Correlations between melody and lyric),)))-)))) )))!7 )-.)))))!7 ) !.#)))!7) .)- #)) ) !#))) ) -)- ))) ))! -))) -)))) )))!)'()) ) )( '(
)
)
,
)
,
)
,
⁄
⁄ ⁄ 7 ⁄ 7 7 7 7 77 7 7))-! )#)!)#))-) !)#)))) !)))!#)) ))#)-)))) ) ) -!)!)!)) )-))!))) ) )!)'!))) )) )') )'! ) )) )) )In) this section, we examine two phenomena re!
)
)
)
)
)
)
)
)
)
)
)
)
)
,
,
,
)!7 . )!7 . lated to boundaries of lyrics: (1) the positions of⁄⁄ ⁄ ⁄ (I felt7when7 you7 warmly7 lit the7 gap7 in my- heart)segment boundaries are biased to melody!)) )) )! )))))! #))#)) )) #)))!)!)#)) ))!)) )!) )! )-)-))7 !) ))-!!#))- )-)7 )!)))!))!).!#))-) ))!)) ).) )!) .)-)lyrics
)
!
#)
!
#)
!
)
)
)
)
)
)
)
)
)
rest
positions,and (2) the probability of boundary)⁄⁄ ⁄ ⁄
7 7 7 occurrence
7
on the duration of a rest, i.e.,! )) ) #)! ))! )) !!#)-) ))#)))!))!! ) -)) ! )) ()!))! )#) ))) ) #)))!( )#)), ))!) !) ))!) )-,) ))7) ) ! -))!! )7#))- ))7 ! )#))))!! )#))-))) ) -) ) -a))shorter restdepends
)
!
)
)
tends
to
be a word boundary and a77
7 first time
7 I noticed your lovely smile) 7 7 7(The⁄⁄ ⁄ ⁄
rest tends to be a block boundary, as shown!) ) )7) ! )') !)7 ) )7 ) ) ) longer
)
)
)
)
)
!#)
)
!#)
!
)
)
)!
)!
!
)
)
!
)
)
!
)!
)
)!
)!
)!
!
)
!
)
)
)
)
)
)
)
)
(
)
)
)
(
(
(
(
!
!
)
)
(
(
)
'
)
)
)
,
,
,
)
)
in
Figure3. All analyses were performed on the
,
,
7
7
7
7
7
⁄ 3: Example boundaries appearing immediately⁄⁄ ⁄ Figure
training
splitof the melody-lyrics alignment data,
after
a rest.
BOB
indicatesa )blockboundary.
!#)
)
!#)
!#)
)
)
)
)
)
(
!
)
)
(
(
!
!
)
)
(
)
'
(
(
)
)
)
'
'
)
)
)
)
)
)
,
,
,
7 7 7
which is described in Section 2.
⁄⁄ ⁄ ⁄For the first phenomenon, we first calculated
the distribution of boundary appearances at the po⁄⁄ ⁄ ⁄sitions of melody notes and rests. Here, by theboundary of a line (or block), we refer to the po⁄⁄ ⁄ ⁄sition of the beginning of the line (or block). In
Figure 3, we say, for example, that the boundary
⁄ ⁄⁄

Fl Fl
139

Fl

Fl Fl
1713

Fl

Fl Fl
2117

Fl

Fl Fl
2521

Fl

Fl Fl
2925

Fl

Fl Fl
3329

Fl

Fl Fl
3733

Fl

Fl Fl
4137

Fl

Fl Fl
4541

Fl

Fl Fl
4945

Fl

45

Fl
Fl
45
49

Fl

Fl

Fl

49

49

49

9

13

17

21

25

29

33

37

41

Fl
Fl
9
13

Fl
Fl
13
17

Fl
Fl
Rest#1
<BOL>
17
21

Fl
Fl
21
25

Rest#2[te- ra-shi][te](light) (FUNC)FlFl Rest#4
25
29

<BOB>

Word boundary Rest#3 Word boundary[ku-re][ta] [to- ki] [ka-n-ji] [ta] [no](FUNC)(FUNC)(when) (feel) (FUNC)(FUNC)
Rest#5

Word boundary[ha-ji- me-te] [ki-zu- i][ta] [hi][no][a-na-ta] [no](first) (notice)(FUNC)(time)(FUNC) (you) (FUNC)

Fl
Fl
29
33

Fl
Fl
33
37

Fl
Fl
37
41

Fl
Fl
41
45

5

(MIDI Tick)

Figure 4: Distribution of the number of boundaries in
the melody-lyrics alignment data.

scores from online forum sites3 ; we thus created
1,000 melody-lyrics pairs. We refer to these 1,000
melody-lyrics pairs as a melody-lyrics alignment
data4 and refer to the remaining 53,181 lyrics without melody as a raw lyrics data.
We randomly split the 1,000 melody-lyrics alignments into two sets: 90% for analyzing/training
and the remaining 10% for testing. From those,
we use 20,000 of the most frequent words whosesyllable counts are equal to or less than 10, andconverted others to a special symbol hunknowni.
All of the digital music score data we collected
were distributed in the UST format, a common fileformat designed specifically for recently emerging
computer vocal synthesizers. While we focus on
Japanese music in this study, our method for data
creation is general enough to be applied to otherlanguage formats such as MusicXML and ABC,
because transferring such data formats to UST is
straightforward.
3

For selecting the 1,000 songs, we chose only frequently
downloaded or highly popular songs to ensure the quality of
the resulting dataset.
4
We publicly release all source URLs of the 1,000 songs(https://github.com/KentoW/melody-lyrics).

of the first block beginning “te-ra-shi te” coincides
with Rest#1. The result, shown at the top of Figure 4, indicates that line and block boundaries arestrongly biased to rest positions and are far lesslikely to appear at note positions. Words, lines, andblocks rarely span beyond a long melody rest.
The bottom of Figure 4 shows the detailed distributions of boundary occurrences for different
durations of melody rests, where durations of 480
and 1920 correspond to a quarter rest and a whole
rest, respectively. The results exhibit a clear, strong
tendency that the boundaries of larger segments
tend to coincide more with longer rests. To thebest of our knowledge, this is the first study thathas ever provided such strong empirical evidence
for the phenomena related to the correlations between lyrics segments and melody rests. It is also
important to note that the choice of segment boundaries looks like a probabilistic process (i.e., there
is a long rest without a block boundary). This observation suggests the difficulty of describing thecorrelations of lyrics and melody in a rule-based
fashion and motivates our probabilistic approach
as we present in the next section.

4

Melody-conditioned language model

Our goal is to build a language model that generates
fluent lyrics whose discourse segment fit a given
melody in the sense that generated segment boundaries follow the distribution observed in Section 3.
We propose to pursue this goal by conditioning a
5

The beginning of a line/block and the end of a line/blockare equivalent since there is no melody between the end andbeginning of a line/block.

165𝑤#$) 𝑠#$)
Output ha-shi-ri
3
Word (run)𝑤#$% 𝑠#$%sa-ru(away) 2

𝐲&#$)

#$)
𝐲"

#$%
𝐲"

𝐲&#$%

𝑤#

𝑠#

<BOL>

0

#
𝐲"

long rest. This means, for example, that when theincoming melody sequence for a next word position is note, note, (long) rest, note, note, as thesequence following to mi(t−1) in Figure 5, it is desirable to select a word whose syllable count is twoor less so that the generated word does not overlap
the long rest. If there is sufficient data available,
this tendency may be learned directly from the correlation between rests and words without explicitly
considering the syllable count of a word. However,
our melody-lyrics alignments for 1,000 songs are
insufficient for this purpose.
We take two approaches to address this data sparsity problem. First, we propose two training strategies that increase the number of training examples
using raw lyrics that can be obtained in greater
quantities. Second, we construct a model that predicts the number of syllables in each word, as well
as words themselves, to explicitly supervise thecorrespondence between rest positions and syllable
counts.
In the following sections, we first describe thedetails of the proposed model and then present thetraining strategies used to obtain better models with
our melody-lyrics alignment data.

Syllable
Count

𝐲&#

LSTM
Word 𝐯(𝑤#$- ) 𝐱 #$)(Embeddingka-ra(from)
𝑤#$- 𝐧#$)Context
MelodyRepresentation

𝐱 #$%
(

𝐯(𝑤#$% )𝐱 #(ha-shi-ri(run)𝑤#$) 𝐧#$%sa-ru(away)𝑤#$%𝐧#𝐯(𝑤#$) )⁄ @@ , - )2 ) )) ) ) ) ) ) ) )! - ⁄) @@) #), -) )2 )(!)) ) ) , ) ) ) ) )! - ) ) #) ) (! ,⁄ , - )2 )…) ) ) ) ) ) ) ) ) , )⁄ )…, )- ))2 )…(!) ) ) ) , ) ) ) ) ) , ) ) ) ) (! … ,⁄ , - )2 ) ) ) ) ) ) ) ) ) )! - ⁄) ), #)- ) 2 ) (!) ) ) ) , ) ) ) ) )! - ) ) #) ) (! ,)⁄ , - )2 ) ) ) ) ) ) ) ) ) ) , )⁄ ), )- ))2 ) (!) ) ) ) , ) ) ) ) ) , ) ) ) (! ,Figure#) ) RNNLM.
#) 5: Melody-conditioned))) ) -))))) (⁄ , - )7 ) ) ) ) )!7 . ) ) ) ) - )7 ⁄) ), )- ))7 ) ( ) )!7 . )
7
!
!
)
)
#
)
)
# ) ) ) ) - ) )!with)) ') ) ) ) - )7 RNNLM) ) ) ) ' ) ) - )7 ) input) - )7 )! ) melody.
⁄ , ) standard
7 ⁄ , ) a )featurized
We call))!.! ) )) ) ) #) ) )! ) ) ) )RNNLM.
! )this! - )!) ) ) )!. ) )) ) #) model) )! ) ) )⁄)a! Melody-conditioned)!)))⁄ The) )network
7
structure of the
7 model is illustrated
!
)
)
!
!
#)
!
#)
#
)
)
)
)
)
)
)
)
)
)
!
#)
#
)
)
5.
Formally,
we
are
given
a
melody))!
)⁄ ! ) )in#) ) Figure
)
)
)
)
)
)
)
)
)
)
) - )7) ) )! ) ) )⁄) - 7 ) - )) -7
7m = m1 ,...,mi ,...,mI that is a sequence of notes)! )7 ) ) )) ) )!)! - )7 )!) ) )7a (pitch) and) )! - rests,)! )7 ( where)⁄ !) ) and) , )m⁄! ) !)) includes, )! ) )a) duration information. Our model generates lyrics w =) ) ) ) ) 'and) ⁄) )( ) is) , 'a) )sequence(! - )7 of)! # ) words(⁄ w, 1) ,...,w) (! t,...,w- )7 T)! # ) thatsegment boundary symbols: hBOLi and hBOBi,a line and a block bound- 4.1 Model construction⁄⁄ special symbols denoting
ary, respectively. For each time step t, the model
The proposed model is based on a standard⁄
outputsasinglewordorboundary
symboltakinga⁄
RNNLM (Mikolov et al., 2010):
pair of the previously generated word wt−1 and theQ⁄ nt for the current word posi⁄ musical feature vector
P (w) = Tt=1 P (wt |w0 , ..., wt−1 ),(1)1

𝑚1 #$% $%4

5

9

1

TARGETNOTE

𝑚1 #$% 3%4

𝑚1 #$% $- 𝑚1(#$%) 𝑚1 #$%5 3ha- shi- ri
(run)
𝑤#$)

𝑚1 # $%4

𝑚1 # $-TARGET
NOTE

𝑚1 # 3%4

𝑚1(#) 𝑚1 # 3-

ha- shi- ri sa-ru(run)(away)
𝑤#$)
𝑤#$%
Context Melody Vector
9

13

13

17

17

21

21

25

25

29

29

33

33

37

37

41

41

45

45

49

49

tion which includes context window-based featuresthat we describe in the following section. In thismodel, we assume that the syllables of the generated words and the notes in the input melody havea one-to-one correspondence. Therefore, the position of the incoming note/rest for a word positiont (referred to as a target note for t) is uniquely determined by the syllable counts of the previously
generated words.6 The target note for t is denoted
as mi(t) by defining a function i(·) which maps
time step t to the index of the next note in t.
Here, the challenging issue with this model istraining. Generally, language models require alarge amount of text data to learn well. Moreover,this is also the case for learning correlation between
rest positions and syllable counts. As shown in Figure 4, most words are supposed to not overlap awhere context words are encoded using
LSTM (Hochreiter and Schmidhuber, 1997)
and the probabilities over words are calculated
by a softmax function. w0 = hBi is a symbol
denoting the beginning of lyrics. We extend thismodel such that each output is conditioned by
the context melody vectors n1 , ..., nt , as well as
previous words:
P (w|m) =t=1 P (wt |w0 , ..., wt−1 , n1 , ..., nt ).

(2)

The model simultaneously predicts the syllable counts of words by sharing the parametersof LSTM with the above word prediction modelin order to learn the correspondence between the
melody segments and syllable counts:
P (s|m) =6

Note that our melody-lyrics alignment data used in training does not make this assumption, but we can still uniquely
identify the positions of target notes based on the obtained
melody-word alignment.

QT

QT

t=1 P (st |w0 , ..., wt−1 , n1 , ..., nt ),(3)

where s = s1 , ..., sT is a sequence of syllable
counts, which corresponds to w.

166

Algorithm 1 Pseudo melody generation

For each time step t, the model outputs a word
distribution ytw ∈ RV and a distribution of syllable
count yts ∈ RS using a softmax function:
ytw = softmax(BN(Ww zt )),(4)

yts = softmax(BN(Ws zt )),(5)

1: for each syllable in the input-lyrics do
2:b ← get boundary type next to the syllable
3:
sample note pitch p ∼ P (pi |pi−2 , pi−1 )4:sample note duration dnote ∼ P (dnote |b)5:
assign note with (p, dnote ) to the syllable
6:
sample binary variable r ∼ P (r|b)
7:
if r = 1 then8:
insert rest with duration drest ∼ P (drest |b)
9:
end if
10: end forwhere zt is the output of the LSTM for each time
step. V is the vocabulary size and S is the syllable
count threshold.7 Ww and Ws are weight matrices. BN denotes batch normalization (Ioffe andSzegedy, 2015).
The input to the LSTM in each time step t is aconcatenation of the embedding vector of the previous word v(wt−1 ) and the context melody representation xtn , which is a nonlinear transformationof the context melody vector nt :xt = [v(wt−1 ), xtn ],

(6)

xtn

(7)

= ReLU(Wn nt + bn ),(MIDI Tick)

Figure 6: Distribution of the number of boundaries in
pseudo-data.

where Wn is a weight matrix and bn is a bias.
To generate lyrics, the model searches for theword sequence with the greatest probability (Eq.
2) using beam search. The model stops generating
lyrics when the syllable count of the lyrics reaches
the number of notes in the input melody.
Note that our model is not specific to the language of lyrics. The model only requires the sequences of melody, words, and syllable counts anddoes not use any language-specific features.
4.2

7

4.3

The syllable counts of the hBOLi and hBOBi are zero.
We rounded each duration to one of the values 60, 120,
240, 360, 480, 720, 960, 1200, 1440, 1680, 1920, and 3840
and use one-hot encoding for each rounded duration.

Training strategies

Pretraining The size of our melody-lyrics alignment data is limited. However, we can obtain alarge amount of raw lyrics. We, therefore, pretrain
the model with 53,181 raw lyrics and then finetune it with the melody-lyrics alignment data. In
pretraining, all context melody vectors nt are zero
vectors. We refer to these pretrained and fine-tuned
models as Lyrics-only and Fine-tuned models, respectively.

Context melody vector

In Section 3, we indicated that the positions ofrests and their durations are important factors formodeling boundaries of lyrics. Thus, we collecta sequence of notes and rests around the current
word position (i.e., time step t) and encode theirinformation into context melody vector nt (see thebottom of Figure 5).
The context melody vector nt is a binary feature vector that includes a musical notation type(i.e., note or rest), a duration8 , and a pitch for eachnote/rest in the context window. We collect notes
and rests around the target note mi(t) for the current word position t with a window size of 10 (i.e.,mi(t)−10 , ..., mi(t) , ..., mi(t)+10 ).
For pitch information, we use a gap (pitch interval) between a target note mi(t) and its previous
8

note mi(t−1) . Here, the pitch is represented by a
MIDI note number in the range 0 to 127. For example, the target and its previous notes are 68 and65, respectively, and the gap is +3.

Learning with pseudo-melody We propose amethod to increase the melody-lyrics alignment
data by attaching pseudo melodies to the obtained
53,181 raw lyrics. We refer to the model that uses
this data as the Pseudo-melody model.
Algorithm 1 shows the details of pseudo-melody
generation. For each syllable in the lyrics, we firstassign a note to the syllable by sampling the probability distributions. The pitch of each note is generated based on the trigram probability. Then, we determine whether to generate a rest next to it. Since
we established the correlations between rests andboundaries of lyrics in Section 3, the probability fora rest and its duration is conditioned by a boundary167

type next to the target syllable. All probabilities arecalculated using the training split of the melodylyrics alignment data.
Figure 6 shows the distributions of the numberof boundaries in the pseudo data. The distributions
closely resemble those of gold data in Figure 4.

5

Model
Lyrics-only
Full-data
Alignment-only
Heuristic
Fine-tuned
Pseudo-melody(w/o ys )Fine-tuned
Pseudo-melody
Human

Quantitative evaluation

We evaluate the proposed Melody-conditioned
RNNLMs quantitatively based on two evaluation
metrics: (1) a test set perplexity for measuring thefluency; (2) a line/block boundary replication task
for measuring the consistency between the melody
and boundaries in the generated lyrics.
5.1

5.2

Evaluation metrics

Perplexity Test-set perplexity (PPL) is a standard evaluation measure for language models. PPL
measures the predictability of wording in original lyrics, where a lower PPL value indicatesthat the model can generate fluent lyrics. We
used PPL and its variant PPL-W, which excludes
line/block boundaries, to investigate the predictability of words.
Accuracy of boundary replication Under theassumption that the line and block boundaries ofthe original lyrics are placed at appropriate positions in the melody, we evaluated consistency
between the melody and boundaries in the generated lyrics by measuring the reproducibility of theboundaries in the original lyrics. Here the metric
we used was F1 -measure of the boundary positions.
We also asked a person to place line and blockboundaries at plausible positions for randomly selected 10 input melodies that the evaluator has

BOB
0.121
0.122
0.298
0.373
0.260
0.318

155.1
118.0
-

0.318
0.312
0.717

278.1
201.5
-

F1 -measure
BOL
UB
0.061 0.106
0.063 0.108
0.287 0.477
0.239 0.402
0.302 0.479
0.241 0.406
0.241
0.250
0.671

0.366
0.406
0.751

Table 1: Results of the quantitative evaluation. “UB”
denotes the score for unlabeled matching of line/blockboundaries. “w/o ys ” denotes the exclusion of thesyllable-count output layer.

never heard. This person is not a professional musician but an experienced performer educated onmusicology. The bottom part of Table 1 represents
the human performance.

Experimental setup

In our model, we chose the dimensions of the wordembedding vectors and context melody representation vectors to 512 and 256, respectively, and thedimension of the LSTM hidden state was 768. We
used a categorical cross-entropy loss for outputs
ytw and yts , Adam (Kingma and Ba, 2014) with an
initial learning rate of 0.001 for parameter optimization, and a mini-batch size of 32. We applied
an early-stopping strategy with a maximum epoch
number of 100, and training was terminated after
five epochs of unimproved loss on the validation
set. For lyrics generation, we used a beam searchwith a width of 10. An example of the generated
lyrics is shown in the supplemental material.

Perplexity
PPL
PPL-W
138.0
225.0
135.9
222.1
173.3
314.8
175.8
284.7
152.2
275.5
115.7
197.5

5.3

Effect of Melody-conditioned RNNLM

To investigate the effect of our language models,
we compared the following six models. The firstone is (1) a Lyrics-only model, a standard RNNLM
trained with 54,081 song lyrics without melody information. The second and third ones are baseline
Melody-conditioned RNNLMs where the proposedtraining strategies are not applied: (2) a Full-data
model trained with mixed data (54,081 song lyrics
and 900 melody-lyrics alignments of those), and(3) an Alignment-only model trained with only 900
melody-lyrics alignment data. The fourth one is astrong baseline to evaluate the performance of the
proposed approaches: (4) a Heuristic model that(i) assigns a line/block boundary to a rest based on
its duration with the same probability, as reported
in Figure 4, and (ii) fills the space between any
two boundaries with lyrics of the appropriate syllable counts. This Heuristic model computes thefollowing word probability:
P (wt |w0 , ..., wt−1 , m) =(8)
Q(hBOBi|mi(t+1) )(if wt = hBOBi)





(if wt = hBOLi)Q(hBOLi|mi(t+1) )(1 − Q(hBOBi|mi(t+1) ) − Q(hBOLi|mi(t+1) ))×
PLSTM (wt |w0 ,...,wt−1 )
1−PLSTM (hBOLi|w0 ,...,wt−1 )−PLSTM (hBOBi|w0 ,...,wt−1 )(otherwise)

where Q is the same probability as reported in Figure 4. PLSTM is the word probability calculated bya standard LSTM language model. The remaining
two are Melody-conditioned RNNLMs with theproposed learning strategies: (5) Fine-tuned and(6) Pseudo-melody models.

168

Lyrics in test set(MIDI Tick)

Lyrics generated by Heuristic model(MIDI Tick)

Lyrics generated by Pseudo-melody model(MIDI Tick)

Figure 7: Distribution of the number of boundaries in the test set and lyrics generated by the Heuristic and Pseudomelody models.

The top part of Table 1 summarizes the performance of these models. Regarding the boundary
replication, the Heuristic, Alignment-only, Finetuned, and Pseudo-melody models achieved higher
performance than the Lyrics-only model for unlabeled matching of line/block boundaries (i.e., UB).
This result indicates that our Melody-conditioned
RNNLMs successfully capture the consistency between melody and boundaries of lyrics. The results of the Full-data model is low (as expected)
because the size of the melody-lyrics alignment
data is far smaller than that of the raw lyrics data
and this harms the learning process of the dependency between melody and lyrics. For the blockboundary, the Heuristic model achieved the best
performances. For the line boundary, the Finetuned model achieved the best performances.
Regarding PPL and PPL-W, the Lyrics-only,
Full-data, and Pseudo-melody models show better results than the other models. The Fine-tuned
model shows reduced performance compared with
the Lyrics-only model because fine-tuning witha small amount of data causes overfitting in thelanguage model. Also, the training size of the
Alignment-only model is insufficient for learning alanguage model of lyrics. Interestingly, the Pseudomelody model achieved better performance than the
Full-data model and overall achieved the best score.
This result indicates that the Pseudo-melody model
uses the information of a given melody to make abetter prediction of its lyrics word sequence. On
the other hand, the Heuristic model had the worst
performance, despite training with a large amountof raw lyrics. We analyze the reason for such performance and describe our results in Section 5.5.
It is not necessarily clear which to choose, either
the Fine-tuned or Pseudo-melody model, which
may depend also on the size and diversity of the
training and test data. However, one can conclude

at least that combining a limited-scale collectionof melody-lyrics alignment data with a far larger
collection of lyrics-alone data boosts the model’s
capability of generating a fluent lyrics which structurally fits well the input melody.
5.4

Effect of predicting syllable-counts

To investigate the effect of predicting syllablecounts, we compared the performance of the proposed models to models that exclude the syllablecount output layer ys . The middle part of Table 1
summarizes the results. For the pretraining strategy, the use of ys successfully alleviates data sparsity when learning the correlation between syllable
counts and melodies from only words themselves.
As can be seen, the model without ys shows reduced performance relative to both PPLs and theboundary replication. On the other hand, for thepseudo-melody strategy, the two models are competitive in both measures. This means that the
Pseudo-melody model obtained a sufficient amountof word-melody input pairs to learn the correlation.
5.5

Analysis of melody and generated lyrics

To examine whether the models can capture correlations between rests and boundaries of lyrics, we
calculate the proportion of the word, line, and blockboundaries in the original lyrics and in the lyrics
generated by the Heuristic and Pseudo-melody
model for the test set (Figure 7). The proportionof hBOLi and hBOBi generated by the Heuristic
model are almost equivalent to those of the originallyrics. On the other hand, for the Pseudo-melody
model, the proportion of line/block boundary types
for the longer rests are smaller than that of theoriginal lyrics.
Although the Heuristic model reproduces the
proportion of the original line/block boundaries,
the model had a low performance in terms of PPL,

169

N
600
600
500
500
400
400
300
300
200
200
100
100
0
0

200

NumberNumberofofblocks
blocksNumberNumberofoflines
lines

300

JS-divergence: 0.075

100

0
0

0
10
10 0

JS-divergence: 0.172

20

30

40

20 5
30 10
40 15
Syllable20
counts30
per line40
counts per
per lineline
Syllablecounts

Lyrics in test set

50
5020
50

25

30
30
25
25
20
20
15
15
10
10
5
5
0
0

JS-divergence: 0.144
JS-divergence: 0.190

00 30

Lyrics generated by Heuristic model

50
50
50

100
100
100

150
150
150

35
40
45
50
Syllablecountsper
block
Syllable counts
counts per
per blockblock
Syllable

200
200
200

Lyrics generated by Pseudo-melody model

Syllable counts per line
Figure 8: Distribution of the syllable count of the generated lines/blocks
Measure
L
G
LM
DM
OQ

Heuristic
Means ± SD Median
2.06±1.08
22.28±1.07
22.34±1.07
22.33±1.10
22.01±1.01
2

Lyrics-only
Means ± SD Median
2.33±1.23
22.81±1.16
3
2.91±1.15
32.80±1.06
32.59±1.15
3Fine-tuned
Means ± SD Median
2.85±1.20
32.79±1.06
32.70±1.13
3
2.59±1.1132.42±1.08
2

Pseudo-melody
Means ± SD Median
2.93±1.14
3
2.97±1.08
32.96±1.09
32.89±1.07
32.65±1.01
3

Human (Upper-bound)
Means ± SD Median
3.56±1.33
43.50±1.25
43.49±1.35
43.49±1.30
43.32±1.19
4

Table 2: Results of the qualitative evaluation.

as shown in Section 5.3. By investigating the lyrics
generated by the Heuristic model, we found that themodel tends to generate line/block boundaries after
the melody rest, even if the two rests are quite close.
Figure 8 shows the distributions of the syllable
per line / block frequency and the distributions of
the Jensen-Shannon divergence. While the Heuristic model tends to generate short lines/blocks, ourmodel generates the lyrics so that lines/blocks do
not become too short. This result supports that (i)our model is trained using melody and lyric contexts and (ii) the heuristic approach, which simply
generates line/block boundaries based on the distribution in Figure 4, cannot generate fluent lyrics
with well-formed line/block lengths.

6

Qualitative evaluation

To asses the quality of the generated lyrics, inspiredby (Oliveira, 2015), we asked 50 Yahoo crowdsourcing workers to answer the following five questions using a five-point Likert scale:
Listenability (L) When listening to melody andlyrics, are the positions of words, lines, and segments natural? (1=Poor to 5=Perfect)
Grammaticality (G) Are the lyrics grammaticallycorrect? (1=Poor to 5=Perfect)
Line-level meaning (LM) Is each line in the lyrics
meaningful? (1=Unclear to 5=Clear)
Document-level meaning (DM) Are the entire
lyrics meaningful? (1=Unclear to 5=Clear)
Overall quality (OQ) What is the overall qualityof the lyrics? (1=Terrible to 5=Great)

For the evaluation sets, we randomly selected four melodies from the RWC Music
Database (Goto et al., 2002). For each melody,
we prepared four lyrics generated by the Heuristic,
Lyrics-only, Fine-tuned, and Pseudo-melody models. Moreover, to obtain an upper bound for thisevaluation, we used the lyrics created by amateur
writers: we asked four native Japanese speakers towrite lyrics on the evaluation melody. One writer
was a junior high school teacher of music who had
experience in music composition and writing lyrics.
Three writers were graduate students with different
levels of musical expertise. Two of the three writers
had experience with music composition, but noneof them had experience with writing lyrics.9 As aresult, we obtained 50 (workers) × 4 (melodies) ×
5 (lyrics) samples in total. We note that workers did
not know whether lyrics were created by a humanor generated by a computer.
Table 2 shows the average scores, standard deviations, and medians for each measure. Regarding the“Listenability” evaluation, workers gave high scoresto the Fine-tuned and Pseudo-melody models thatare trained using both the melody and lyrics. This
result is consistent with the perplexity evaluation
result. On the other hand, regarding the “Grammaticality” and “Meaning” evaluation, workers gavehigh scores to the Lyrics-only and Pseudo-melody
models that are well-trained on a large amount oftext data. This result is consistent with the result of170

9We release lyrics and audio files used in the qualitative evaluation on the Web (https://github.com/
KentoW/deep-lyrics-examples).

the boundary replication task. Regarding the “Overall quality” evaluation, the Pseudo-melody model
outperformed all other models. These results indicate our pseudo data learning strategy contributes
to generating high-quality lyrics. However, thequality of lyrics automatically generated is stillworse than the quality of lyrics that humans produce, and it still remains an open challenge for
future research to develop computational models
that generate high-quality lyrics.

7

Related work

In the literature, a broad range of research efforts
has been reported for computationally modelinglyrics-specific properties such as meter, rhythm,
rhyme, stress, and accent Greene et al. (2010);
Reddy and Knight (2011); Watanabe et al. (2014,2016). While these studies provide insightful findings on the properties of lyrics, none of those takes
the approach of using melody-lyrics parallel data
for modeling correlations of lyrics and melody
structures. One exception is the work of Nichols
et al. (2009), who used melody-lyrics parallel data
to investigate, for example, the correlation between
syllable stress and pitch; however, their exploration
covers only correlations at the prosody level but
not structural correlations.
The same trend can be seen also in the literatureof automatic lyrics generation, where most studies utilize only lyrics data. Barbieri et al. (2012)
and Abe and Ito (2012) propose a model for generating lyrics under a range of constraints provided in terms of rhyme, rhythm, part-of-speech,
etc. Potash et al. (2015) proposes an RNNLM
that generates rhymed lyrics under the assumption that rhymes tend to coincide with the end oflines. In those studies, the melody is considered
only indirectly; namely, input prosodic/linguistic
constraints/preferences on lyrics are assumed tobe manually provided by a human user because
the proposed models are not capable of interpreting and transforming a given melody to constraints/preferences.
For generating lyrics for a given melody, we
have so far found in the literature two studies
which propose a method. Oliveira et al. (2007)
and Oliveira (2015) manually analyze correlations
among melodies, beats, and syllables using 42 Portuguese songs and propose a set of heuristic rules
for lyrics generation. Ramakrishnan A et al. (2009)attempt to induce a statistical model for generating

melodic Tamil lyrics from melody-lyrics parallel
data using only ten songs. However, the former captures only phonological aspects of melody-lyrics
correlations and can generate a small fragment oflyrics (not an entire lyrics) for a given piece of
melody. The latter suffers from the severe shortageof data and fails to conduct empirical experiments.

8

Conclusion and future work

This paper has presented a novel data-driven approach for building a melody-conditioned lyrics
language model. We created a 1,000-song melodylyrics alignment dataset and conducted a quantitative investigation into the correlations between
melodies and segment boundaries of lyrics. No
prior work has ever conducted such a quantitative
analysis of melody-lyrics correlations with this sizeof data. We have also proposed a RNN-based,
melody-conditioned language model that generates fluent lyrics whose word/line/block boundaries
fit a given input melody. Our experimental results have shown that: (1) our Melody-conditioned
RNNLMs capture the consistency between melody
and boundaries of lyrics while maintaining word
fluency; (2) combining a limited-scale collection of
melody-lyrics alignment data with a far larger collection of lyrics-alone data for training the model
boosts the model’s competence; (3) we have also
produced positive empirical evidence for the effectof applying a multi-task learning schema where
the model is trained for syllable count prediction as
well as for word prediction; and (4) the human judgments collected via crowdsourcing showed that ourmodel improves the quality of generated lyrics.
For future directions, we plan to further extend
the proposed model for capturing other aspects oflyrics/melody discourse structure such as repetitions, verse-bridge-chorus structure, and topicalcoherence of discourse segment. The proposed
method for creating melody-lyrics alignment data
enables us to explore such a broad range of aspectsof melody-lyrics correlations.

Acknowledgments
This study utilized the RWC Music Database (Popular Music). This work was partially supported
by a Grant-in-Aid for JSPS Research Fellow Grant
Number JP16J05945, JSPS KAKENHI Grant Numbers JP15H01702, and JST ACCEL Grant Number
JPMJAC1602. The authors would like to thank Dr.
Paul Reisert for the English language review.

171

References
Chihiro Abe and Akinori Ito. 2012. A Japanese lyrics
writing support system for amateur songwriters. In
Proceedings of Asia-Pacific Signal & Information
Processing Association Annual Summit and Conference 2012 (APSIPA ASC 2012). pages 1–4.
Dave Austin, Jim Peterik, and Cathy Lynn Austin.
2010. Songwriting for Dummies. Wileys.
Gabriele Barbieri, François Pachet, Pierre Roy, andMirko Degli Esposti. 2012. Markov constraints forgenerating lyrics with style. In Proceedings of the20th European Conference on Artificial Intelligence(ECAI 2012). pages 115–120.
Masataka Goto, Hiroki Hashiguchi, Takuichi
Nishimura, and Ryuichi Oka. 2002. RWC Music Database: Popular, classical and jazz music
databases. In Proceedings of the 3rd International
Conference on Music Information Retrieval (ISMIR
2002). volume 2, pages 287–288.
Erica Greene, Tugba Bodrumlu, and Kevin Knight.
2010. Automatic analysis of rhythmic poetry withapplications to generation and translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP 2010).
pages 524–533.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long
short-term memory. Neural computation 9(8):1735–
1780.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In Proceedingsof the 32nd International Conference on Machine
Learning (ICML 2015). volume 37, pages 448–456.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint
arXiv:1412.6980.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto.
2004.
Applying conditional random fields toJapanese morphological analysis. In Proceedings ofthe 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP 2004). pages
230–237.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan
Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent
neural network based language model. In Proceedings of Interspeech 2010. pages 1045–1048.
Saul B. Needleman and Christian D. Wunsch. 1970.
A general method applicable to the search for similarities in the amino acid sequence of two proteins.
Journal of Molecular Biology 48(3):443–453.
Eric Nichols, Dan Morris, Sumit Basu, and Christopher Raphael. 2009. Relationships between lyrics
and melody in popular music. In Proceedings of the10th International Society for Music Information Retrieval Conference (ISMIR 2009). pages 471–476.

172

Hugo Gonçalo Oliveira. 2015. Tra-la-lyrics 2.0: Automatic generation of song lyrics on a semantic domain. Journal of Artificial General Intelligence
6(1):87–110.
Hugo R. Gonçalo Oliveira, F. Amialcar Cardoso, andFrancisco C. Pereira. 2007. Tra-la-lyrics: An approach to generate text based on rhythm. In Proceedings of the 4th International Joint Workshop on
Computational Creativity. pages 47–55.
Peter Potash, Alexey Romanov, and Anna Rumshisky.
2015. GhostWriter: Using an LSTM for automatic Rap lyric generation. In Proceedings of the2015 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2015). pages 1919–
1924.
Ananth Ramakrishnan A, Sankar Kuppan, andSobha Lalitha Devi. 2009. Automatic generationof Tamil lyrics for melodies. In Proceedings of theWorkshop on Computational Approaches to Linguistic Creativity. pages 40–46.
Sravana Reddy and Kevin Knight. 2011. Unsupervised
discovery of rhyme schemes. In Proceedings of the
52nd Annual Meeting of the Association for Computational Linguistics (ACL 2011). pages 77–82.
Tatsuji Ueda. 2010. よくわかる作詞の教科書 The
writing lyrics textbook which is easy to understand(in Japanese). YAMAHA music media corporation.
Kento Watanabe, Yuichiroh Matsubayashi, Kentaro
Inui, and Masataka Goto. 2014. Modeling structural
topic transitions for automatic lyrics generation. In
Proceedings of the 28th Pacific Asia Conference on
Language, Information and Computation (PACLIC
2014). pages 422–431.
Kento Watanabe, Yuichiroh Matsubayashi, Kentaro
Inui, Tomoyasu Nakano, Satoru Fukayama, andMasataka Goto. 2017. LyriSys: An Interactive support system for writing lyrics based on topic transition. In Proceedings of the 22nd International
Conference on Intelligent User Interfaces (ACM IUI2017). pages 559–563.
Kento Watanabe, Yuichiroh Matsubayashi, Naho Orita,
Naoaki Okazaki, Kentaro Inui, Satoru Fukayama,
Tomoyasu Nakano, Jordan Smith, and Masataka
Goto. 2016. Modeling discourse segments in lyrics
using repeated patterns. In Proceedings of the 26th
International Conference on Computational Linguistics (COLING 2016). pages 1959–1969.