NLP4MusA 2021

Atypical Lyrics Completion Considering Musical Audio Signals
Kento Watanabe and Masataka Goto, National Institute of Advanced Industrial Science and Technology (AIST)◆ Lyrics Completion Task
We propose a lyrics completion task to recommend candidate words for a blank in a given sentence.

INPUT
The [ blank ] in my heart.

OUTPUT
The [ pain ] in my heart.
The [ place ] in my heart.
The [ deep ] in my heart.
The [ fire ] in my heart.
…

Lyrics Completion
System1

What Words Should be Suggested?
◆ Lyrics Completion Task
Suggesting words that are (1) atypical but (2) suitable for the musical audio signals.
1. Atypicality is important in writing lyrics in a creative way; to make lyrics attractive, lyricwriters usually consider both typical and atypical phrases.
However, previous research on automatic lyrics generation has used language models
LSTM-LM,GPT2,GPT3,
that predict highly frequent (i.e., typical) words.

L The pain in my heart.

Transformer…

J The agony in my heart

Many songs use this phrase…

This is unique!

2. Lyrics depend on the moods of music.
(e.g., “love” is often used in ballad songs, “kill” is often used in metal songs).

➔ … love you …

➔ … kill you …
Metal song

Ballad song

2

Previous Studies and Our Study◆ Previous study of general word completion task
Generating highly frequent (i.e., typical) words.

•OUTPUTINPUT
The [ blank ] in my heart.

The [ pain ] in my heart.
The [ place ] in my heart.
The [ deep ] in my heart.
…

Language model

Draft

Typical words◆ This study
Suggesting words that are (1) atypical but (2) suitable for musical audio signals.

•INPUT
The [blank] in my heart
INPUTandOUTPUTThe [ agony ] in my heart.
The [ dagger ] in my heart.
The [ plague ] in my heart.
…

Lyrics Completion
SystemDraft and audio

Atypical words
3Demo System

4Key Idea•

We propose a multimodal vector space model: Lyrics-context2vec.
1. We encode both of the input draft sentence with a blank and audio signals of the song into an
input vector, and also encode a lyric word appropriate for the blank into a lyric word vector.
2. We train the model to make those vectors located near each other.
INPUT
The [blank] in my heart

Make closerandInput vector

Draft and audio
Positive
Exampleagony
Lyric word
vectorLyric word

Multimodal vector space
5Key Idea•

We propose a multimodal vector space model: Lyrics-context2vec.
3. A highly frequent word is encoded into a typical word vector.
4. We use it as a negative example to make it located far away from the input vector.
INPUT
The [blank] in my heart

Make closer

andInput vector

Draft and audio
Positive
Exampleagony
Lyric word
vectorLyric word
Negative
ExamplePush awaypain
Typical word
vectorHighly frequenttypical word

Multimodal vector space
6

Key Idea•

We propose a multimodal vector space model: Lyrics-context2vec.
5. Only atypical words are suggested since they are close to the input vector of a draft sentence
and audio signals.

INPUT
The [blank] in my heart

OUTPUT

Make closer

andInput vector

agony
daggerplague…

Lyric word
vectorAtypical words that aresuitable for input.

Draft and audio
Positive
Exampleagony
Lyric word

Negative
ExamplePush awaypain
Typical word
vectorHighly frequent
typical word

Multimodal vector space
7

Training Lyrics-context2vec
We used contrastive learning by extracting positive and negative pairs from a large-scale dataset.

INPUT

458,572 songs

Input vector

Draft and audio

agony
Lyric wordvector

co-occurring word

pain

Push away

Typical word
vectorHighly frequent
typical word

Training data

Make closer

Contrastive Learning

The agony in my heart…

Positive pair

Negative pair

Lyrics and Audio signal

The [blank] in my heartandMultimodal vector space

Positive and negative pairs

8Model Construction◆ Lyrics-context2vec

Make closer

Lyric word vector

Typical word vectorPush away

Embedding

Embedding

Lyricscorpusw’wt
Input vector

Randomly sample from corpus.

Concatenation

Left draft vector

Song-level audio vector

Right draft vector

LSTM

Averaging

Embedding

Embedding

…

…

w2

w1

wT-1

wt+1

wt-1
…

All

Pseudo words aw1

wT…

you needis faith in…

awM

Signal to word conversion

myheart.

Input lyrics text and audio signal
9Signal to word conversion◆ Bag of Audio-words
To embed audio signals, we use a discrete symbol called an audio-word.

Audio signals① wav② MFCCs+flatten

250ms 250ms 250ms 250ms 250ms 250ms 250ms 250ms 250ms 250ms 250ms③ k-means

k=1

k=1

k=2

k=2

k=3

k=3

k=4

k=3

k=1

k=5

④ audio-words

aw1

aw2

aw3

aw4

aw5

aw6

aw7

aw8

aw9

aw10 aw11

k=5

audio-words
The audio can be used with NLP techniques by converting the signal to symbols.
10Experiments (1)• We calculated how many of the top 20 words suggested by each model were typical or atypical.
• We calculated the document frequency of words and assumed that the top 10% of them are
typical words (i.e., the remaining 90% are atypical words).

RNN-language model

Vector space model (VSM)• VSMs suggested many more atypical words than RNN-LM type models did.
• Contrastive learning contributes to suppression of typical word completion while RNN-LM typemodels maximize the probability of predicting typical words.
11Experiments (2)We evaluated whether the proposed method could fill in the blanks of the test lyrics with theoriginal words.
As words to fill in the blanks in the test
lyrics, we chose words that wereatypical and fit the mood of the music.
Test data
INPUT
The [blank] in my heart

Lyrics and Audio signal

The agony in my heart…

OUTPUT
Proposedmethod

232,304 songs

The [ plague ] in my heart.
The [ dagger ] in my heart.
The [ plague ] in my heart.
…

In the top N suggested words, if the original word issuggested, the evaluation score is 1, (i.e., 0 otherwise).

12

Experiments (2)We evaluated whether the proposed method could fill in the blanks of the test lyrics with theoriginal words.

• Lyrics-context2vec predicted atypical words suitable for music audio better than allother models.
• Our model captures both the atypicality and the relationship between a music audio andwords simultaneously.
13Examples
Input draft sentence: The [blank] in my heart.
Input Audio: Killing time (Metal song)
Method: LSTM-based Encoder-Decoder (previous approach)pain, deep, burning, fire, hole, world, dead, darkness, feeling, beauty, devil, silence,drowning, shadows, words, dream, demons, power, wind, thunder
Method: Lyrics-context2vec (our approach)hole, devil, burning, emptiness, blackness, pain, darkness, demons, dagger, hatred,tremors, void, fire, agony, holes, essence, coldness, plague, needles, deep
• Our lyrics-context2vec can successfully suggest some rare words in bold fonts, such as"dagger" and "agony".
• Lyrics-context2vec successfully suggested explicit and negative words, such as"darkness", "agony", and "coldness". Those words are suitable for the mood of the
input metal song.
14Conclusion◆ Task• We propose a lyrics completion task to recommend candidate words for a blank in agiven sentence.
• Our task aims to suggest words that are (1) atypical and (2) suitable for the musicalaudio signal.

◆ Method• We proposed lyrics-context2vec, a multimodal vector space model that suggests
atypical but appropriate words for the given music audio and draft sentence.
• Input vector and output word vector located near each other.
• Vectors of highly frequent word located far away from the input vector.

◆ Findings• Our contrastive learning strategy contributes to suggesting atypical words.
• Embedding audio signals contributes to suggesting words suitable for the mood of theprovided music audio.
15