Combining Learned Lyrical Structures andVocabulary for Improved Lyric Generation

Pablo Samuel Castro
Google Brain
psc@google.com

Maria AttarianGoogle
jmattarian@google.com

Abstract
The use of language models for generating lyrics and poetry has received anincreased interest in the last few years. They pose a unique challenge relative tostandard natural language problems, as their ultimate purpose is creative; notionsof accuracy and reproducibility are secondary to notions of lyricism, structure,and diversity. In this creative setting, traditional quantitative measures for naturallanguage problems, such as BLEU scores, prove inadequate: a high-scoring model
may either fail to produce output respecting the desired structure (e.g. songverses), be a terribly boring creative companion, or both. In this work we propose amechanism for combining two separately trained language models into a frameworkthat is able to produce output respecting the desired song structure, while providing
a richness and diversity of vocabulary that renders it more creatively appealing.

1

Introduction

With the increased realism and sophistication of generative models, artists have been increasingly
drawn to incorporate these methods into their creative process. The approaches vary, from transferring
style from one artist to another (Dumoulin et al., 2016) to adapting a pre-existing process to produce
abstract art that maximizes the likelihood of a category under a classification model (White, 2018).
Lyrics are a particularly challenging artistic endeavour; high-quality lyrics typically require following
a specific lyric structure, the use of a rich vocabulary, a mastery of the language, and the use of poetic
techniques such as metaphors and alliteration. Because of this, the use of machine learning models
for the generation of lyrics has seen a slower increase. The few cases where machine learning models
have been used for lyric generation have required a substantial amount of human intervention. In oursubmission to the Machine Learning and Creativity Workshop at NIPS 2017 (Castro et al., 2017) wetrained a Recurrent Neural Network (RNN) over a dataset of lyrics. We then manually curated thelyrics produced with renowned Canadian songwriter David Usher to rewrite one of his songs (Sparkleand Shine). Although a successful experiment in human-machine collaboration, the lyrics required
more manual intervention than we would have liked. We recently switched to the more sophisticatedTransformer Language Models (TLMs) (Vaswani et al., 2017) to train over the same dataset. The
results are of substantially improved, but although they seem to maintain the general structure oflyrics, they still suffer from a lack of variety.

2

Proposed Framework

Our approach combines two different TLMs. The first model (LS ) is trained to capture the structureof lyrics, while the second (LV ) is trained to provide a richer vocabulary than what is currently
available in the lyrics dataset, while still leveraging the context of the existing lyrics. Given an initial
input lyric l1 , we combine these models to produce the next line as follows (P oS will be describedbelow): l2 = RichLyrics(l1 ) ≡ LV (l1 · LS (P oS(l1 ))).
32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.

LS : Our dataset consists of a large set of lyrics spanning multiple genres and decades (see Appendix A). Our inputs consist of the separate lines of all the song lyrics, while the targets are thesame lines shifted by one (e.g. linen → linen+1 ). We pre-processed the lyrics by converting theminto their respective Parts-of-Speech (PoS)1 . This was done to ensure that the Lyric model is only
capturing lyric structure, but not vocabulary. We will refer to this conversion process as P oS(l); inother words our input-to-target mapping becomes P oS(linen ) → P oS(linen+1 ).
LV : We picked a subset of Project Guttenberg’s Top 20 books Kaggle dataset2 (the list of books used
is provided in Appendix B). We split each sentence s into two parts: s1 and s2 ; the splitting point waschosen at about halfway through the sentence, without splitting any words (see Appendix C for moredetails). Denoting · as string concatenation, a sentence s is converted to an input-to-target mappingas: s1 · P oS(s2 ) → s2 . The intuition behind this approach is that s1 provides the context, while
P oS(s2 ) provides the structure to be “materialized”.

3

Empirical Evaluation

In order to evaluate our approach we generated 100 lyric verses using the following procedure.
We randomly picked 100 lines from our lyrics dataset as starter lines l1 . Then, for each model
we incrementally built a verse of 5 lines by setting li+1 ← RichLyrics(li ). We are using beam
search with max size 3, so each li results in 3 different li+1 s. We consider the verse produced byeach of these possibilities. This means that for each l1 we produce up to 35 = 243 different verses(depending on the input, the beam search may sometimes produce less than 3 variants). We compared
our RichLyrics approach against two baselines: P ureLyrics is a TLM trained only on the lyrics
dataset; P ureBooks is a TLM trained only on the books dataset.
3.1

Quantatitative Evaluation

From the verses generated by each model we computed the number of words and average word
length per line, the number line repeats in the verse (li ≡ li+1 ), and the fraction of words repeatedfrom one line to the next. The results are presented in Table 1 and demonstrate that RichLyrics
makes use of a much larger vocabulary and with fewer repeats. Given that the verses are 5 lines
long, P ureLyrics is repeating lines about half the time! Qualitative examples of the generations arepresented in Appendix D and further confirm these quantitative results.
Words per line
P ureLyrics
P ureBooks
RichLyrics

4

Word length
Number of line
per line
repeats
12.853 ± 1.106 3.351 ± 0.060 2.233 ± 0.125
6.633 ± 0.024
3.758 ± 0.010 0.002 ± 0.001
6.928 ± 0.046
3.542 ± 0.016 0.441 ± 0.039
Table 1: Statistics for the different models.

Fraction ofrepeated words
0.671 ± 0.026
0.244 ± 0.002
0.480 ± 0.009

Discussion and Future WorkAlthough we are able to substantially improve the quality of the generated lyrics, there is still muchwork ahead of us. We would like to train over a larger set of books, and ones that are more current
to have more modern vocabulary. An important aspect of lyric structure that we are investigatingis having the generation adapt to rhyming structure and phonetic cadence, as this is something
songwriters use often to fit a musical melody. As with most language models out there, semanticconsistency still proves challenging, and is something we are actively investigating.

1
2

We used pos_tag from Python’s nltk library to extract PoS.
https://www.kaggle.com/currie32/project-gutenbergs-top-20-books

2

References
Castro, P. S., Usher, D., and Larochelle, H. (2017). Sparkle and shine 2.0. https://vimeo.com/274650906.
Dumoulin, V., Shlens, J., and Kudlur, M. (2016). A learned representation for artistic style. CoRR,
abs/1610.07629.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., andPolosukhin, I. (2017). Attention is all you need. CoRR, abs/1706.03762.
White,T.
(2018).
Perception
Engines.
https://medium.com/
artists-and-machine-intelligence/perception-engines-8a46bc598d57.

ALyrics dataset

These are the genres used for the lyric structure model detailed in Section 2. We exclude Children’sMusic and Hip-Hop, the latter to reduce the amount of profanity in the generations.
• Alternative / indie• Country• Folk• Jazz• Metal• Pop• R-and-B / Soul• Rock• Soundtracks
In total this resulted in over 1 million input/target pairs, and about 91,000 for the test/validation sets.

BKaggle books dataset

The books used for training the Lyric Vocabulary model detailed in Section 2 were:• A Tale of Two Cities by Charles Dickens• Adventures of Huckleberry Finn by Mark Twain• Alices Adventures in Wonderland by Lewis Carroll• Dracula by Bram Stoker• Emma by Jane Austen• Frankenstein by Mary Shelley• Great Expectations by Charles Dickens• Grimms Fairy Tales by The Brothers Grimm• Metamorphosis by Franz Kafka• Pride and Prejudice by Jane Austen• The Adventures of Sherlock Holmes by Arthur Conan Doyle• The Adventures of Tom Sawyer by Mark Twain• The Count of Monte Cristo by Alexandre Dumas• The Picture of Dorian Gray by Oscar Wilde• The Prince by Nicolo Machiavelli• The Yellow Wallpaper by Charlotte Perkins Gilman
In total this resulted in around 155,000 input/target pairs used for training, and around 13,000 for thetest/validation sets.
3

C

Word splitting mechanism

The split happens assuming that periods denote the end of a sentence (aside from abbreviations
typically containing a period e.g. Mr. or St.) while also taking into account that quotes in books
sometimes mark the end of a sentence, e.g. in the case of quotes being followed by an uppercase
word. In order to avoid very long sentences skewing the structure of the proposed framework, weare splitting into subsentences of 15 words each, if the total word number in a sentence exceeds athreshold. Subsequently, we split each sentence into approximately half, in respect of not cutting
words in half. This has been parameterized in case we choose to experiment with different ratios.
As an example, if we have the sentence: “The quick brown fox jumped over the fence”, our procedure
would produce the following input and target phrases:• Input: The quick brown fox VBD IN DT NN• Target: jumped over the fence

4

D

Qualitative Evaluation

We present some sample lyrics produced by the different models, using the same starter lines (initalics below). As discussed in Section 2, we generate lines incrementally: l1 → l2 → · · · ln . In
Table 2 we compare P ureLyrics with RichLyrics, where the increased variety in outputs producedby RichLyrics is evident.
Table 2: Comparison between P ureLyrics and RichLyrics models.
PureLyrics
come on , uh
i’m not gonna write you a love song’cause you tell me it’s
i’m not the one you wanted
i’m not the one you wanted
i’m not the one you wanted
i ’m your big and brave and handsome romeo
i’m the man of the woods, i’m the man of the woods
i’m the man of the woods, i’m the woods
i’m the man of the woods, i’m the woods
i’m the man of the woods, i’m the woods
i’m the man of the woods, i’m the woods
you told me you loved nobody else , you never would
you told me you loved nobody else but you
you told me you loved nobody but you
you told me you loved nobody but you
you told me you loved nobody but you
you told me you loved me but you loved me

RichLyrics
come on , uh
you remember the voice of the widow
i love the girl of the age
i have a regard for the wholei have no doubt of the kindi am sitting in the corner of the mantelpiece
i ’m your big and brave and handsome romeo
you know my secret secret
you have my second estate
you suit your high origin
you have my cursed youth
you have my life
you told me you loved nobody else , you never would
you told me you wanted everything else, you never would
he put his hand on the pillow of the marquis
he put his cap on the ground like a stone
he put his hand on the latch of a door
he put his key in the lock as a key

In Table 3 we compare P ureBooks with RichLyrics, which highlights how our proposal produces
output that is more reminiscent of real lyrics, both in terms of phrase structure and length.
Table 3: Comparison between P ureBooks and RichLyrics models.
PureBooks
come on , uh
she was a new man
but it was nota thing to be done
it wasa confession
i ’m your big and brave and handsome romeo
you, and i’ll tell you all about it
i don’tunderstand you,said the young man, and weshall be happy to-day
you told me you loved nobody else , you never would
have felt that you were coming to know of yourself
i have no
doubt of that,said the young man, that you have been agreat fancy for a few minutes, and then another?

5RichLyrics
come on , uh
you remember the voice of the widow
i love the girl of the age
i have a regard for the wholei have no doubt of the kindi am sitting in the corner of the mantelpiece
i ’m your big and brave and handsome romeo
you know my secret secret
you have my second estate
you suit your high origin
you have my cursed youth
you have my life
you told me you loved nobody else , you never would
you told me you wanted everything else, you never would
he put his hand on the pillow of the marquis
he put his cap on the ground like a stone
he put his hand on the latch of a door
he put his key in the lock as a key