Text-to-Lyrics Generation with Image-based Semantics
and Reduced Risk of Plagiarism 【ISMIR2023】
Kento Watanabe and Masataka Goto.

National Institute of Advanced Industrial Science and Technology■ Question: How to check if lyrics are plagiarized?
J Phrases that are safe to generate:Ø New phrases that do not exist in the dataset.
Ø Commonly used phrases, such as “I love you”.

■ Goal: Text-to-lyrics generation system by suggesting
the generated lyrics to users who struggle to find theright words to convey their message.
J Desirable System

Ø Generate lyrics that conveya message similar to theinput text.
Ø Generate lyrics as a sourceof new inspiration.

L Undesirable System

Ø Generate lyrics that are
similar in wording to theinput text.
Ø Generate lyrics that may notprovide enough inspiration.

Input Text

Input Text

Driving a car on the seaside

Driving a car on the seaside

Text-to-Lyrics Generation

Text-to-Lyrics Generation

Generated Lyrics

Generated Lyrics

I 'm driving in my car
Just like a fish on an ocean floor

Driving a car along the coastline

The wording information fromthe input text is removed.

Ø But, we must train the Image-to-Lyrics Enc-Dec:
L To train our Enc-Dec, image-lyric data is required.
J We therefore generate images from existing lyrics byusing a diffusion model to create training data.

likeafish… 〈/P〉

Softmax Activation
Image Features(50 768 dimension)…

Fully Connected Layer

Transformer Decoder(Trainable Parameters)

Pre-trained Vision Transformer(Freezed Parameters)
Patch + Positional
Embedding0※(※Extra embedding)

12

3

4

…

46

47

48

49

Word + Positional
Embedding0(T 768 dimension)… 456

Linear Projection of Flattened Patches

Image(512 512 pixels)…
Sequence of image patches
Resize the image to 224 224 pixels and divide it intopatches of 32 32 pixels each, for a total of 49 patches.

7892…

1016

Embedding Layer〈P〉…

inmycar〈L〉

JustlikeLyrics (Context)a…

■ Quantitative Evaluation:
Perplexity (PPL): Predictability of the phrasing in theoriginal lyrics in the test set.
Normalized Edit Distance (NED): Evaluate whether theproposed method generates lyrics that differ inwording from the input text.
EnglishJapaneseTraining paired data for Enc-Dec:
MethodPPL ↓ NED ↑PPL ↓ NED ↑[X]-to-Lyrics Generation
84.86
346.73544.21163.98

0.78
0.69
0.71
0.68

231.49306.191051.58
583.13

0.92
0.86
0.66
0.90.

To
evaluate
theproposed
lyricsgeneration
method,wecon20%Englishstructed a small test dataset consistingof pairs Japaneseof lyrics and15%
input text representing the semantic content of the lyrics.
10%
Since such 57.28%
a dataset
totalis not available, for English songs, we
5%prepared a test dataset
54.31% that
total included plot texts from 20 Dis0%
neyanimatedfilms,
takenfromWikipedia,
alongwith
their
12
3
4
56
7
8
9
10
1112
N song lyrics. We here assume that thecorresponding theme
L Without ourlyrics
proposedanti-plagiarism
method,of each theme song are written based on the content
approximately
57.3%
ofthegenerated
N-grams(whereof that film. For Japanese songs, we prepared 51 Japanese
N ranges from 1 to 12) matched our UncommonPhrase,animation plot texts and their theme song lyrics.
Percentage of generated
N-grams included in
UncommonPhrase.

Lyrics
Just

We're so bright and shining
On our way to the ice■ Ablation Test for the Anti-Plagiarism Method■ Model Structure: A model that connects a pre-trained
Vision Transformer with a Transformer Decoder.
〈L〉

Come to me, come to me
I want a little rabbit andit's the same as you

4.1 Experimental dataset

927.5K generated images/
1,08M generated images

carGenerated Lyrics

J Image-lyric pairs are more effective than other pairTable 1. Resultsof quantitativeevaluation.
ed data sets as training
data forEnc-Dec generating
lyrics that are semantically related to the input textbut differ from it in wording.

Pre-trained
Diffusion model:
Anything-v3

my→ Image →I2L (proposed)
Image-to-Lyrics(proposed)
Summary-text-to-Lyrics
S2L
Back-translated-lyrics-to-Lyrics
B2L
First-half-lyrics-to-Second-half-lyrics
H2H

Text-to-Image Generation…

Input Text (plot summary)J We succeeded in using images to capture moods ofthe input text and generating lyrics accordingly.

Ø Advantages of using the intermediate image:- The image represents the semantics of the input text
without holding its wording information.
- The style of the input text can be any word, phrase,sentence, or paragraph.

On■ Examples of Generated Lyrics

I 'm driving in my car
Just like a fish on anocean floor

Only the semantics of theinput text are conveyed.

927.5K English lyric paragraphs /1,08M Japanese lyric paragraphs…

Image-to-Lyrics
Encoder-Decoder

To test our lyrics generation, we used plot summaries
of 20 Disney films from Wikipedia and their theme song
lyrics, assuming the lyrics reflect the film's content.

Generated Lyrics

Generated Image

Pre-trained
Driving a car
Diffusion Modelon the seasideCheck

In a park in England, a young girl named
Alice with her cat, Dinah, listens distractedly to her sister's history lesson, andbegins daydreaming of a nonsensical world. She spots a passing White Rabbit…
Frozen. Princess Elsa of Arendellepossesses magical powers allowing her tocontrol ice and snow, often using them toplay with her younger sister Anna. AfterElsa accidentally injures Anna …

■ Key Idea: Generating lyrics from input text by routing
through an intermediate image.
Input Text…

■ Contributions:(1) Propose a two-step pipeline that enables training ofa text-to-lyrics generation system even withoutpaired text-lyrics data.
(2) Propose a lyric generation method that reduces therisk of plagiarizing lyrics from the training data.

Lyrics in the process of generation
UncommonPhrase
N-gramDF• Like a fish on an ocean…
RETAIN• And the land is dark …
the night has come 2
DISCARDRETAINthe land is dark
1 Phrase • This is the last time…
…

■ Technical Problem:(1) Training a text-to-lyrics encoder-decoder is notfeasible since there is no text-lyric paired data.
(2) Machine learning methods may unintentionallyreproduce training data (plagiarize existing lyrics).

L Phrases with a risk of plagiarizing existing lyrics:
Ø Uncommon phrases used in only a few songs.
■ Proposed Method:(1) Collect all word N-grams with DF (DocumentFrequencies) 1-3 in the training data and add them
to a list of uncommon phrases, UncommonPhrase.
(2) During the lyrics generation process, if any phrases
within the generated lyrics are found in UncommonPhrase, those lyrics are discarded.

indicating the risk of plagiarism.
J Our method effectively reduces this risk of plagiarism
Comparedfrom 57.3%4.2
to Methods
0%.