IUI 2017 • Recommender Systems

March 13–16, 2017, Limassol, Cyprus

LyriSys: An Interactive Support System for Writing Lyrics
Based on Topic Transition
Kento Watanabe1 , Yuichiroh Matsubayashi1 , Kentaro Inui1 , Tomoyasu Nakano2 ,Satoru Fukayama2 , Masataka Goto2
1 Graduate School of Information Sciences, Tohoku University, Japan
2 National Institute of Advanced Industrial Science and Technology (AIST), Japan{kento.w, y-matsu, inui}@ecei.tohoku.ac.jp, {t.nakano, s.fukayama, m.goto}@aist.go.jp
Step1): Set musical structure (The number of lines/blocks/syllables)

ABSTRACTThis paper presents LyriSys, a novel lyric-writing support system. Previous systems for lyric writing can fully automatically
only generate a single line of lyrics that satisfies given constraints on accent and syllable patterns or an entire lyric. In
contrast to such systems, LyriSys allows users to create andrevise their work incrementally in a trial-and-error manner.
Through fine-grained interactions with the system, the user
can create the specifications of the musical structure and thestory of the lyrics in terms of the verse-bridge-chorus structure,
the number of lines, words and syllables, and most importantly,
the transition over semantic topics such as hscenei, hdarki andhsweet lovei. This paper provides an overview of the design ofthe system and its user interface and describes how the writing
process is guided by a state-of-the-art probabilistic generative
topic model that is trained without supervision. The system
works for both Japanese and English.

Step2): Set topic transition

Generated lyrics

Selected topic

Setting Panel

Edit Panel

Generation Button
Step3): Generate lyrics

Figure 1. An example LyriSys screen.

ACM Classification Keywords

H.5.2. Information Interfaces and Presentation (e.g., HCI):
Natural language; H.5.2. Information Interfaces and Presentation (e.g., HCI): Training, help, and documentation.
Author Keywords

Computational creativity; linguistic creativity; song lyrics;
semantic; topic transition.
INTRODUCTION

In popular music, lyrics are an important element in conveying
emotions and messages. The recent increasing popularity of
consumer generated media makes writing lyrics increasingly
popular even for novice users. In writing lyrics, the writerconsiders various desirable factors, as follows [2, 12]:Syllable In the case of writing lyrics for a given piece of music, the writer must select words whose syllables correspondto the melody notes.
Rhyming and refrain Lyrics have their own specific properties such as frequent repetitions of identical or similarphrases and extensive use of rhymes and refrains.

Story Popular music tends to be structured by components
called “verses”, “bridges” and “choruses”. In this paper,
we refer to such components as blocks. Each piece oflyrics corresponding to a block tends to conveys a particular
topic such as hscenei or hsweet lovei, and the manner inwhich one topic transition to another is important in writing
coherent lyrics. In this paper, we refer to the sequence of
topics in a song as a story.
For writers of lyrics, however, considering these factors simultaneously is not easy. This is where appropriate humancomputer interaction system can reduce human loads. This
paper presents our novel Japanese and English lyric-writing
support system, “LyriSys” (Figure 1). LyriSys can assist awriter in incrementally taking the above factors into accountthrough an interactive interface by generating candidate piecesof lyrics that satisfy the specifications provided by the writer.
The capability of automatically generating lyrics and allowing the user to create lyrics incrementally in a trial-and-error
manner can be useful for both novices and experts.
RELATED WORK

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IUI 2017, March 13–16, 2017, Limassol, Cyprus.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-4348-0/17/03 ...$15.00.
http://dx.doi.org/10.1145/3025171.3025194

Writing support

How a computer system can support writing is an intriguingquestion. Existing support systems can be classified into two
types, systems that can generate entire lyrics automatically[9, 10, 8] and tools designed to assist the user in searching
for words that satisfy a query and usage examples from stored

559

IUI 2017 • Recommender Systems

March 13–16, 2017, Limassol, Cyprus

Overview of the edit panel
LyriSys generates lyrics at the checked areaa

Set the topic of each blockbe Change the number of lines/blocks

New Line is added

d

c

After

BeforeClick the button

New block is added

Click the button
Directly edit modea Uncheck the check box

b Change the number of syllables

Before generating the lyrics

uncheckede
Set the number of syllables

Directly edit

After generating the lyrics

LyriSys does not regenerate lyrics at the unchecked area
The lyrics that specify the changed syllable

Set the number of lines/blocks

Figure 2. The edit panel in LyriSys; the user can set the musical structure, the number of lines/blocks/syllables ( b and e ), and the user is allowed todirectly edit the current draft using the keyboard ( c ).

lyrics.1 However, we believe that neither type of system isappropriate for lyric-writing support. The former type allows
users highly limited interaction, whereas the latter requires
users to do everything but word search.
GarageBand2 is a system commonly used by beginners as asupport tool for song composition. In this interface, the user
selects abstract conditions such as hcheerfuli and hdarki, andthen the interface searches for melody patterns that satisfy the
specified conditions, thereby enabling novice users to compose
a song incrementally by only selecting and setting out the
melody pattern with a time line. However, there is no lyricwriting interface for the user to input abstract conditions. In
LyriSys, it is relatively easy to create lyrics that represent a
story by selecting a topic, such as hscenei or hsweet lovei.
Automatic lyric generation

Prior studies report systems that can generate only a single lineof lyrics independently of the rest [1] but none of the previous
systems can consider the global coherence of lyrics such as
the topics of blocks and the similarity between and transitions
over blocks. Several research groups have proposed lyric
generation systems that are designed to generate an entire lyric
fully automatically [3, 11]. However, such fully automatic
systems are not appropriate for the purpose of writing support;
they provide no functionality for the user to control the writing
process interactively. In contrast, we implement an interface
that allows users to specify what they want and consider the
global coherence of lyrics incrementally and interactively.

Step 1) Input the parameters for specifying the musical structure (the number of blocks, the number of lines in eachblock, and the number of syllables in each line) manuallyon the edit panel.
Step 2) For each block, choose a topic from the predefined setof topics manually. LyriSys can also estimate the topics ofa given lyrics automatically. This function enables users tolearn the concept of story and how it works from examples
for, say, their favorite existing lyrics. With this function, theuser can use the system also to fill only a small number oflines of a given mostly completed lyric.
Step 3) Click the Generation button, LyriSys then generates
the lyrics that correspond to the input syllables and the topic.
For example, LyriSys automatically generates “I believe inlove” when the user inputs the syllable set “1-2-1-1” andthe topic hsweet lovei. This function assists the user insearching the huge space of word sequences. Moreover,users can revise some lines, if they desire, by selecting aline to revise and then choosing an alternative candidatefrom the candidates displayed on the Setting Panel.
Along with this process, LyriSys generates candidate lyrics
that satisfy the topic and the number of syllables. A crucial
property of the design of LyriSys is that it allows the user tospecify the constraints incrementally and explore the candidate
phrases interactively in a trial-and-error manner.
Step 1): Set the musical structure

By clicking on the edit panel (Figure 2 e ), the user sets the musical structure, the number of lines and the number of blocks.
The user can also always change the number of syllables foreach line (Figure 2 b ). Changing these parameters is allowed
at any time; therefore the user can flexibly revise the musicalstructure. For example, the user might want to change the musical structure of the second verse slightly, while maintaining
the musical structure of the first verse.

LYRISYS: AN INTERACTIVE WRITING INTERFACE BASEDON GLOBAL STRUCTURE

This section provides an overview of how a user of LyriSys
writes lyrics by interacting with the system. Figure 1 shows ascreen shot of LyriSys’s user interface which consists of theEdit Panel and the Setting Panel. The Edit Panel displays thecurrent specifications of the musical structure and the current
candidate lyrics the user is editing. The Setting Panel is used
to choose topics. The basic process is as follows:Moreover, the user can also disable the regeneration of a particular line by unchecking the check box (Figure 2 a ). Thisfunction allows users to specify the lines they are alreadysatisfied with and seek alternative candidates only for the remaining lines. By gradually disabling the regeneration, theuser can complete the writing process incrementally.

1 MasterWriter. http://masterwriter.com/
2 GarageBand. http://www.apple.com/mac/garageband/

560

IUI 2017 • Recommender Systems

March 13–16, 2017, Limassol, Cyprus

Select a topic of each block
Generated lyrics

Topicedit panel. The user can select a favorite candidate and clickit, resulting in the candidate being displayed in the edit panel.
By setting the parameters and selecting candidates repeatedly,
the user can gradually compose an entire lyric in a trial-anderror manner. The user is also allowed to edit the current draft
directly using the keyboard (Figure 2 c ).

Score

A

B

IMPLEMENTATIONIn this study, to generate lyrics that satisfy the topic and thenumber of syllables, we calculated tri-gram probabilities andadded the number of syllables s and the topic z in the conditions of tri-gram:(0(si , |wi |)P(wi |wi−2 , wi−1 , si , z) = count(z,wi−2 ,wi−1 ,wi )(1)(si = |wi |)count(z,w ,w )DCThe candidate lyrics are displayed
Example words strongly associated with the topic

i−2

i−1

where |wi | denotes the number of syllable in a word wi .
This probability is calculated from count(z, wi−2 , wi−1 , wi ),which returns the number of occurrences of the word string
wi−2 , wi−1 , wi and the topic z. This value is calculated bycounting the number of topics assigned to each block in an
enhanced hidden Markov model (HMM) [13].

Figure 3. The setting panel in LyriSys; the user can set the topic ofeach block that displayed in the edit panel ( A ). LyriSys searches forcandidate lyrics that satisfy the input parameters, and the user can selecta favorite candidate lyrics ( D ).

Step 2): Set/estimate the story

The process of specifying the topics is as follows: (1) the topic
setting panel is displayed (Figure 3) by clicking the topic (e.g.,hscenei) on the edit panel (Figure 2 d ), and (2) the user selects
one topic from the predefined set of topics (Figure 3 A ). The
setting panel displays example words strongly associated withthe topic in question, where the size of each word depicts how
likely it is to be chosen for the specified topic (Figure 3 C ).
This way of displaying the word set is expected to help theuser select topics. LyriSys generates a line of lyrics containing
as many of these words as possible. For example, when the
topic hsweet lovei is chosen, words such as “want” are likely
to be chosen as in “I want you”. It is possible to change thetopics at any time, and the user can revise the generated lyrics.
How our probabilistic language model deals with topics is
described below in the implementation section.
Some writers might have difficulty in selecting topics. We
thus additionally implemented the function of automaticallyestimating the topics of a given lyric (Figure 3 B ). This function is crucially important for users who are not familiar with
the notion of topics. First, by applying this function to existing
popular lyrics (say, their favorite song lyrics) and seeing theresults (i.e., the estimated topics), users can learn what sorts ofphrases tend to be generated for each different topic and what
transitions over topics can be seen in popular lyrics. Second,
users can also begin the writing process by partially rewriting
their favorite lyrics, while keeping the original overall structure and topic transitions. Third, modeling topic transitions
enables the system to propose a smooth transition of topics forgiven partly completed draft lyrics.
Step 3): Generate/edit the candidate lines of lyrics

LyriSys searches for candidate lyrics that satisfy the input
parameters, when triggered by the generation button, and displays the most probable lyrics in the edit panel (Figure 2 c ).
The user can replace the generated lyrics with other candidates
line by line. The candidate lyrics are displayed in the setting
panel (Figure 3 D ) when selecting a line of the lyrics in the561

The enhanced HMM captures the topic transition, which appears in the block structure. For example hscenei → hdarki→ hsweet lovei represents the transition of the topic in threeblocks. In particular, each zt is generated from the previous
topic zt−1 via the transition probability P(zt |zt−1 ). The word
w in each block is generated from zt via generative probability P(w|zt ). In addition, it is possible to estimate the topic
when uncompleted or unknown lyrics are inputted by using theViterbi algorithm. Note that the topic z is not labeled as hsweet
lovei or hscenei, because z is a latent variable. Therefore, we
manually assign labels to each topic by observing the word
list whose generative probability P(w|z) is large. Note that the
number of topics is changeable before learning; however we
set it to ten in this study.
We train the tri-gram and enhanced HMM using unsupervised learning from two datasets that contain Japanese andEnglish lyrics.3 We used 19,290 Japanese lyrics and 96,475
English lyrics and applied the MeCab part-of-speech parser forJapanese words [5] and Stanford CoreNLP for English words[7]. To count the number of syllables, we used a hyphenation
algorithm [6]. In lyrics generation, LyriSys searches the word
strings so that the lyrics probability ∏ni=1 P(wi |wi−2 , wi−1 , si , z)
is large according to the beam search algorithm, where n denotes the number of words in a line.
USER FEEDBACK

To investigate the capabilities, limitations, and potential of ourinteraction design, we asked five Japanese users to use LyriSys
and collected preliminary user feedback. One user was a
junior high school teacher of music who had experience inmusic composition and lyric writing. Four users were graduate
students with different levels of musical expertise. Two ofthem had experience with novel composition, and two had
experience with music composition, but none of them had
experience with lyric writing.
3 http://www.odditysoftware.com/page-datasales1.htm

IUI 2017 • Recommender Systems

MethodBaseline
MethodPrevious
Method 1
Previous
Method 2
Proposed
MethodOverallComments

March 13–16, 2017, Limassol, Cyprus

Positive Comments
In comparison to other tasks, it was comfortable in not specifying the number ofsyllable.
1) It was easy to write the lyrics because I didn’t need to determine which words touse. 2) The system sometimes generated cool lyrics pieces without editing manually.
1) It was easier to write than the baseline method because the system generated
prototype lyrics. 2) It was useful to select the candidate of lyrics when the generated
result was partially good.
1) It was easy to associate the words related to a topic by viewing the generated
candidates of lyrics and the word cloud in the setting panel. 2) In comparison to
the previous method 2, selecting topics made it easy to write the lyrics that specifies
my intention. 3) The generated lyrics are more expressive than the result of otherinterface because of the consideration of topic.
The support interface was helpful to complete the lyrics particularly when I couldn’t
come up with any nice words at all.

Negative Comments
1) It was difficult to come up with the words that satisfy the melody of song,
because of lacking in vocabulary. 2) It was difficult to conceive the story.
I sometimes felt boring because users couldn’t edit the generated lyrics.
It was difficult to write the lyrics that represent the story, because only alimited variety of words are generated.
1) The list of the 10 topics was too restricted and coarse-grained. 2)
Although the system generates an abstract story, I thought that it would
be interesting if the system could generate a concrete story.

1) The speed of automatic generation was slow. 2) I sometimes felt boring
because the generated lyrics were same when input parameters were fixed.
3) I had a hard time inputting the number of syllable manually.

Table 1. Results of user feedback: User’s positive and negative comments.

Block 1, Topic: <情景 (scene)>

Experimental setup

We randomly selected five Japanese songs from the RWC
music database [4] and gave each user one song. Then, we
asked the subjects to write lyrics on the melody of the songwith the following four tasks.

o-mo-i- de no sa-ka-mi-chi wa hi- sa-si-bu-ri no hi- ru sa-ga-ri e

ko- i- bi-to noa- shi-a- to ha a-su-fa-ru-to no a- me ni nu-re-ru

思い出の坂道は久しぶりの昼下がりへ.恋人の足跡はアスファルトの雨に濡れる.

(Afternoon came to the way of memories after a long time. Lovers' footprint get wet in the rain on asphalt.)

思い出の坂道を.雨上がりの交差点で. 思い出の坂道を.心にない雨に濡れて.
(The way of memories. At the intersection of the rain. The way of memories. I got wet in the rain without heart.)

Block 2, Topic: <切ない恋愛 (sweet love)>

Baseline method (without interface) In this task, we restricted the use of LyriSys. We gave the subjects a topic
transition, e.g., hscenei → hsweet lovei → hpositivei, andasked the subjects to write the lyrics that satisfy the given
topic transition. The purpose of this task is to investigate
the difficulty of writing lyrics that satisfy the story and themusical melody without the writing support interface.
Previous method 1 (automatic lyrics generation) We
asked the subjects to write lyrics with LyriSys, but
restricted the use of the selecting/editing of the candidatelines of lyrics. The purpose of this task is to compare theproposed interaction with previous methods that generate
an entire piece of lyrics fully automatically [3, 11].
Previous method 2 (interaction without topic transition)
We implemented another type of LyriSys that has a
restricted topic transition function. This LyriSys calculates
the simple tri-gram probability P(wi |wi−2 , wi−1 , si ). The
purpose of this task is to compare LyriSys with the previous
interface, which cannot handle topic transitions [1].
Proposed method (LyriSys) In this task, we permitted thesubjects to use all of the functions of LyriSys.
Results

After the trial usage, we asked the subjects to write comments
on each task. Positive and negative comments regarding thecapabilities and potential of each task are listed in Table 1.
These comments suggest that the proposed interface is effective, but that the generation algorithm must be improved toenable the user to write more expressive lyrics.
Figure 4 shows an example of lyrics that were created whena user used LyriSys (i.e., the user set the musical structure
and the story, and selected or edited the recommended lyrics).
Moreover, a fully automatically generated lyric is also shownin Figure 4. This result shows that the created lyrics correspond to the input parameters (i.e., syllables and topics); forexample, we can see the sentimental phrases “涙を見せず(not show my tears)” and “あなたのそばに (with you)” werecreated when the topic was hsweet lovei. Note that thesephrases are recommended by LyriSys.

562

se- ka- i- ju- u

de ko-n- na- ni su-re-chi-ga-

ina-mi- da wo mi-se-zu hi-to-ri hu-ru-ma-u da-ke de a-na- tano so- ba ni

世界中でこんなにすれ違い.涙を見せず独り振る舞うだけで.あなたのそばに.

(We could not understand each other in this world. I didn’t show my tears, I was lonely. I just wanna be with you.)

世界中でこんなにたくさんの.心に決めて思い出した思い出に.あなたのそばに.

(So many things in the world. Memories that I remembered in my mind. I just wanna be with you.)

Block 3, Topic: <明るい (positive)>e- i- e- n no hi-ka-ri

ni tsu- tsu-ma-re-ru

ko-to wo

Example outcome ofan user’s interactions
with the system.

i- tsu- ka mi-ta- su

永遠の光に包まれることをいつか満たす.
(Someday I will be filled with eternal light.)
悲しみの涙を忘れれぬように.自分を知る.
(Don’t forget the tears of sorrow. I know myself.)

Fully automaticallygenerated lyrics.

Block 4, Topic: <切ない恋愛 (sweet love)>a-ri- ta-ke no sa-ke-bi-go- e yu- me no ha-ji- ma-ri wo ko- no hi i- da- ki-

yu-

ku

ありったけの叫び声 夢の始まりを この日 抱きゆく.
(Today, I embrace all the screams and the beginning of dream.)
世界中の恋人が.きっと永遠に.この日.ずっとこの.
(Lovers around the world. Surely forever. Today. Forever.)Figure 4. Example of Japanese lyrics when the user uses LyriSys. The
Japanese lyrics are translated, and the English are given in parentheses.
The song is from the RWC Music Database (RWC-MDB-P-2001 No.47).

CONCLUSIONS AND FUTURE WORKIn this paper, we proposed a novel lyric-writing interface,
LyriSys, that allows users to create and revise their work incrementally in a trial-and-error manner. Through fine-grainedinteractions with the system, the user can create the specifications of the music structure (the verse-bridge-chorus structure
and the number of lines/syllables) and the transition over topics such as hscenei, hdarki and hsweet lovei.
LyriSys still leaves much room for improvement. It might
be too much of a burden for the user to specify the musicalstructure at the level of syllable counts. This must be relaxed
possibly by taking the melody’s rhythm directly into account.
The present probabilistic language model models semantic
topics and topic transitions, but not the verse-bridge-chorus
structure, neglecting, for example, the role of choruses. We
plan to fix these problems and improve the system by introducing extended functions on the Web.
ACKNOWLEDGMENTS

This study utilized the RWC Music Database (Popular Music).
This work was supported by a Grant-in-Aid for JSPS Research
Fellow Grant Number JP16J05945, JSPS KAKENHI Grant
Numbers JP15K16045, and CREST, JST.

IUI 2017 • Recommender Systems

March 13–16, 2017, Limassol, Cyprus

Processing Toolkit. In Association for Computational
Linguistics (ACL) System Demonstrations. 55–60.

REFERENCES

1. Chihiro Abe and Akinori Ito. 2012. A Japanese lyrics
writing support system for amateur songwriters. In
Proceedings of Asia-Pacific Signal & Information
Processing Association Annual Summit and Conference2012.

8. Hugo Gonçalo Oliveira. 2015. Tra-la-Lyrics 2.0:
Automatic Generation of Song Lyrics on a Semantic
Domain. Journal of Artificial General Intelligence 6, 1
(2015), 87–110.

2. Dave Austin, Jim Peterik, and Cathy Lynn Austin. 2010.
Songwriting for Dummies. Wileys.
3. Gabriele Barbieri, François Pachet, Pierre Roy, andMirko Degli Esposti. 2012. Markov Constraints for
Generating Lyrics with Style. In Proceedings of the 20th
European Conference on Artificial Intelligence. 115–120.
4. Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura,and Ryuichi Oka. 2002. RWC Music Database: Popular,
Classical and Jazz Music Databases.. In Proceedings ofthe 3rd of International Society for Music Information
Retrieval, Vol. 2. 287–288.

9. Hugo Gonçalo Oliveira, F Amılcar Cardoso, andFrancisco Câmara Pereira. 2007. Tra-la-Lyrics: An
approach to generate text based on rhythm. In
Proceedings of the 4th International Joint Workshop on
Computational Creativity. 47–55.
10. Burr Settles. 2010. Computational creativity tools forsongwriters. In Proceedings of the NAACL-HLTWorkshop on Computational Approaches to Linguistic
Creativity. 49–57.
11. Rajeswari Sridhar, D Jalin Gladis, Kameswaran Ganga,
and G Dhivya Prabha. 2014. Automatic Tamil lyric
generation based on ontological interpretation forsemantics. Jornal of Sadhana 39, 1 (2014), 97–121.

5. Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto.
2004. Applying Conditional Random Fields to JapaneseMorphological Analysis. In Proceedings of the 2004
Conference on Empirical Methods in Natural Language
Processing. 230–237.

12. Tatsuji Ueda. 2010. The writing lyrics textbook which iseasy to understand (in Japanese). yamaha music media
corporation.

6. Franklin Mark Liang. 1983. Word Hy-phen-a-tion by
Com-put-er. Citeseer.

13. Kento Watanabe, Yuichiroh Matsubayashi, Kentaro Inui,and Masataka Goto. 2014. Modeling Structural Topic
Transitions for Automatic Lyrics Generation. In
Proceedings of The 28th Pacific Asia Conference on
Language, Information and Computation. 422–431.

7. Christopher D. Manning, Mihai Surdeanu, John Bauer,
Jenny Finkel, Steven J. Bethard, and David McClosky.
2014. The Stanford CoreNLP Natural Language563