Atypical Lyrics Completion Considering Musical Audio Signals

Kento Watanabe and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST){kento.watanabe, m.goto}@aist.go.jp

Abstract
This paper addresses the novel task of lyrics
completion for creative support. Our proposed
task aims to suggest words that are (1) atypicalbut (2) suitable for musical audio signals. Previous approaches focused on fully automatic
lyrics generation tasks using language models that tend to generate frequent phrases, despite the importance of atypicality for creativesupport. In this study, we propose a novel
vector space model and hypothesize that embedding multimodal aspects (words, draft sentences, and music audio) in a unified vector
space contributes to capturing (1) the atypicality of words and (2) the relationships between
words and the moods of music audio. To test
our hypothesis, we used a large-scale dataset
to investigate whether the proposed model suggests atypical words.

1

Introduction

Lyrics are important in conveying emotions andmessages in popular music, and the recently increasing popularity of user-generated content onvideo sharing services makes writing lyrics populareven for novice writers. Lyrics writers, however,unlike the writers of prose text, need to create attractive phrases suitable for the given music. Writing lyrics is thus not an easy job.
Its difficulty has motivated a range of studies forautomatic lyrics generation (Oliveira et al., 2007;
Potash et al., 2015; Watanabe et al., 2018). For
example, Watanabe et al. train a Recurrent Neural
Network Language Model (RNN-LM) that generates fluent lyrics while maintaining compatibility
between the boundaries of lyrics and melody structures. However, even if LMs generate perfect lyrics,
a fully automatic generation system cannot support
writers because it ignores their intentions.
In this study, for creative support instead of lyrics
generation, we design a lyrics completion task that

Figure 1: Overview of lyrics completion task.

recommends candidate words for the blank in agiven sentence (Fig. 1). Specifically, we focus on
the following two properties of lyrics. (1) Lyrics
sometimes depend on the moods of music audio(e.g., “death” is often used in metal songs) (Watanabe and Goto, 2019). We propose a task in which asystem recommends words suitable for the moodof a given song excerpt represented as an audio signal. (2) Atypicality is important in writing lyrics; tomake lyrics attractive, writers consider both typicaland atypical phrases. However, previous study on
lyrics generation has used LMs that predict highly
frequent (i.e., typical) words (Barbieri et al., 2012;
Potash et al., 2015; Watanabe et al., 2017, 2018).
Creative support systems need to recommend unusual and rare (i.e., atypical) words while maintaining the fluency of the sentence.
We therefore propose a multimodal vector spacemodel (VSM), lyrics-context2vec, that, given adraft sentence with a blank, suggests atypical words
while maintaining the relationship with the moodof the music audio. With lyrics-context2vec, input vectors (i.e., combinations of music audios anddraft sentences) and output vectors (i.e., atypicalwords) are located near each other in a unified highdimensional vector space (Fig. 1). This model suggests atypical words because we use typical words

as negative examples in its training.
The contributions of this study are summarized
as follows: (1) We propose, for creative support, anovel multimodal vector space model that captures
the relationship between atypical words and themood of music audio. (2) We demonstrate that ourmodel suggests words suitable for the mood of theinput musical audio signal. (3) We demonstratethat our model suggests words more atypical thanthose suggested by RNN-LMs.
This paper is a short version of our MMM 2021
paper (Watanabe and Goto, 2021).

2

Lyrics-audio data

To model the relationship between lyrics andmoods of music audio, we obtained 458,572 songs,
each consisting of a pair comprising a text file ofEnglish lyrics and an audio file of a music excerpt1 .
Here each text file contains all sentences of thelyrics of a song, and each audio file is a music
excerpt (30 sec) that was collected from the Internet and provided for trial listening. We embedded
the moods of audio signals as well as the wordsof lyrics into a unified vector space without usingcoarse metadata (e.g., genre tags).
2.1

Bag-of-Audio-Words

To represent the mood feature of a short music
excerpt, we use a discrete symbol called an audioword (Liu et al., 2010). The bag-of-audio-words(BoAW) creation procedure is as follows. (1) Eachmusic excerpt is downsampled to 22,050 Hz. (2)LibROSA, a python package for music and audio
analysis, is used to extract 20-dimensional melfrequency cepstral coefficients (MFCCs) with theFFT size of 2048 samples and the hop size of 512
samples. This result is represented as an MFCC matrix (20 × 1280). (3) The MFCC matrix is divided
into 128 submatrices (20 × 10) without overlap.
(4) To create a vocabulary of k audio-words, weapply the k-means++ algorithm to all the divided
MFCCs of all the songs. In other words, each k-th
cluster corresponds to an audio-word (aw). In thisstudy we made 3000 audio-words.

3

Atypical word completion model

We propose a multimodal vector space model
lyrics-context2vec that, given a music audio signal
and a draft sentence with a blank, suggests atypical words while maintaining the relationship with1

Lyrics were provided by a lyrics distribution company.

the mood of the music audio. Specifically, lyricscontext2vec suggests the best N atypical words
w1 , ..., wN that could fit with the context. Here weassume two types of contexts: (1) the words on theleft and right sides of the blank and (2) the BoAW
converted from the audio signal.
There are two technical problems in recommending atypical words suitable for the music audio.
First, since most LMs learn to predict highly frequent words, it is hard to suggest atypical words
that are important for creative support. Second,
how to model the relationship between words andmusical audio signals is not obvious.
To address the first problem, we focus on thenegative sampling strategy in word2vec (Mikolovet al., 2013). This strategy was proposed for the
purpose of approximation because computation ofloss function is time-consuming. We, however, use
negative sampling for the purpose of suppression of
typical word recommendation because we want tosuggest atypical words for creative support. Since
negative examples are drawn from the distributionof highly frequent words, it is expected that input
vectors of contexts are located far from vectors oftypical words. It is not obvious that the negative
sampling contributes to suggesting atypical words.
To address the second problem, we utilize themechanism of lyrics2vec proposed by Watanabe
and Goto. In lyrics2vec, co-occurring audio-words
and lyric words are located near each other under
the assumption that some words of lyrics are written depending on the musical audio signal.
3.1

Model construction

Lyrics-context2vec is based on lyrics2vec and context2vec (Melamud et al., 2016). Formally, context2vec is a vector space model that encodes
left draft words w1 , ..., wt−1 and right draft words
wt+1 , ..., wT into latent vectors z 1 and z 2 , respectively, using two Recurrent Neural Networks(RNNs). Then the target word vector v(wt ) anda vector that is nonlinearly transformed from thelatent vectors are mapped closely into a unified vector space. The loss function of context2vec Ec2v
is defined so that the inner product of the targetword vector v(wt ) and the nonlinearly transformedvector is maximized:
Ec2v = −logσ v(wt )T · MLP([z 1 , z 2 ])−SXs=1logσ −v(ws0 )T · MLP([z 1 , z 2 ]) ,(1)

where σ(·) is a sigmoid function. To obtain anx-dimensional word vector representation, we define an embedding function v(·) that maps thetarget word to an x-dimensional vector. S isthe number of negative examples ws0 . [z 1 , z 2 ]denotes a concatenation of latent vectors z 1and z 2 . MLP(·) stands for multilayer perceptron. In this loss function, negative examplesws0 are sampled
distribution P (ws0 ) =P from the0 0.75
00.75
D(ws ) / w0 ∈V (D(w ) ) where V is the vocabulary and D(w0 ) is the document frequency ofa word w0 . In other words, since frequent words
tend to be sampled as negative examples, we expectthat a draft sentence vector and the vector of highly
frequent typical words are located far away fromeach other. When computing word completion, oursystem displays target words with high cosine similarity to the input context vector MLP([z 1 , z 2 ]).
Then we extend context2vec to suggest atypicalwords suitable for both the music audio and thedraft sentence by embedding three aspects (i.e., target words, draft sentences, and song-level audio).
We concatenate song-level audio and draft vectors
and define the loss function
E so that the concate1 PM
nated vector [z 1 , z 2 , M
m=1 u(awm )] is locatedclose to the target word vector v(w):M
1 X
E = −logσ v(wt )T · [z 1 , z 2 ,u(awm )]Mm=1−SXlogσ −v(ws0 )T · [z 1 , z 2 ,

s=1

M
1 X

M

u(awm )] ,(2)m=1where we define the dimension of draft vectors
z 1 , z 2 as d and define an embedding function u(·)
that maps the context word/audio-word to a ddimensional vector. M is the number of audiowords in the song. We define the average of audioword vectors as a song-level audio vector.

4Experiments

To evaluate whether lyrics-context2vec can suggest (1) atypical words and (2) words suitable
for music audio, we designed word completion
tasks. The input of these tasks is T − 1 draft
words w1 , ..., wt−1 , wt+1 , ..., wT of each sentence
in a test song. Therefore the model needs to fillin the t-th blank with a word. We used the following Score to evaluate the performance of models
task: Score@N =P in the lyrics 1completion
Nr∈R 1(r ∈ {h , ..., h })/|R|, where r denotes
the correct word and |R| is the number of blanks

in the test data. h1 , ..., hN are the top N suggestedwords. 1(·) is the indicator function. In this study
we calculated Score@N , with N ranging from 1
to 20 under the assumption that our support systemsuggests 20 words to users.
Here it is important to define which word in eachsentence is the correct word r. We defined fourtypes of correct answers:
Typicality We defined a randomly chosen word ineach sentence of the test song as the correct word
r. In this metric, high-frequency words tend to bechosen as the correct answer. In other words, thismetric is a measure of typical word completion.
Atypicality We first calculated the document frequency of words of the test song and then defined
the minimum-document-frequency word in eachsentence as the correct word. This metric is a measure of atypical word completion.
Music+Typicality In each sentence of the test song,
we extracted the word most similar to the music audio of the song by using the pre-trained lyrics2vec
that was proposed by Watanabe and Goto. If thedocument frequency of the extracted word wasmore than 1,000, we defined this word as the correct word for the sentence and did not use the otherwords. This metric is a measure of prediction of
typical words suitable for the music audio.
Music+Atypicality We extracted the word mostsimilar to the music audio of the song as with Music+Typicality. If the document frequency of theextracted word was less than or equal to 1,000,
we defined this word as the correct word for thesentence and did not use the other words. This
metric is a measure of prediction of atypical words
suitable for the music audio of the song.
4.1

Comparison methods

To investigate the effect of our lyrics-context2vec,
we compared the following four models. (1) BiRNN-LM, a bidirectional RNN-LM trained withlyrics without audio. (2) Encoder-Decoder, a BiRNN-LM
in which the song-level audio vector
1 PMu(awm ) is input to the initial RNN state.
m=1M(3) Context2vec (Melamud et al., 2016). (4) Lyricscontext2vec, the proposed model. The RNN-LMs(Bi-RNN-LM and Encoder-Decoder) predict words
with high predicted probability in the blank, and theVSMs (context2vec and lyrics-context2vec) predictthe most similar words in the blank. Examples ofwords suggested by the models are available ata web page (https://kentow.github.io/

Figure 2: Results of the lyrics completion tasks.

nlp4musa2021/).
4.2

Settings

We randomly split our dataset into 80-10-10% divisions to construct the training, validation, and test
data. In all models, we utilized Long Short-TermMemory (LSTM) (Hochreiter and Schmidhuber,1997) as the RNN layer. We chose d = 300 for thedimension of the audio-word vector u(·) and thedimension of the LSTM hidden state z. We chosex = 900 for the dimension of the target word vector v(·). We used negative sampling with S = 20
negative examples. We used Adam (Kingma andBa, 2015) with an initial learning rate of 0.001
for parameter optimization. The model used fortesting was the one that achieved the best Music+Atypicality score on the validation set.

for atypical word completion tasks it is desirable
to use a VSM with negative sampling rather than a
LM aimed at generating typical phrases.
Regarding the main task Music+Atypicality(Fig. 2 (d)), lyrics-context2vec predicted atypicalwords suitable for music audio better than any of
the other models. This means that our model captures both the atypicality and the relationship between a music audio and words simultaneously.
Moreover, we can see that lyrics-context2vec performs better than context2vec and that EncoderDecoder performs better than Bi-RNN-LM. This
indicates that using audio information contributes
to suggesting atypical words suitable for the music
audio.

54.3

Conclusion

Results

Figure 2(a) shows the result of the typical word
completion task (Typicality). As shown in this figure, RNN-LMs achieved higher scores than VSMs.
This is because the RNN-LMs are trained to maximize the probability of generating highly frequentphrases. Interestingly, we can see that there is nodifference between the scores of Bi-RNN-LM andEncoder-Decoder. This indicates that audio information does not contribute to predicting typicalwords. Typical words were thus expected to becorrelated with draft sentences rather than audio.
Regarding the task of predicting the typicalwords suitable for music audio (Fig. 2 (c)), we canobserve results similar to those for the task Typicality. This reinforces the fact that typical words canbe predicted from only the draft sentence, without
using audio information.
Regarding the atypical word completion (Fig. 2(b)), VSMs achieved higher scores than RNN-LMs.
This indicates that negative sampling contributes tosuppression of typical word completion. Overall,

We proposed lyrics-context2vec, a multimodal vector space model that suggests atypical but appropriate words for the given music audio and draft
sentence. In the vector space of lyrics-context2vec,
a vector corresponding to an atypical word in asong and a song-level audio vector corresponding
to an audio excerpt of the song are located neareach other. We trained the models to suggest atypical words by embedding the highly frequent word
vector away from the song-level audio vector.
In the experiment, we used a large-scale dataset
to investigate whether the proposed model suggests
atypical but appropriate lyrics. Several findings
were obtained from experiment results. One is thatthe negative sampling contributes to suggesting
atypical words. Another is that embedding audio
signals contributes to suggesting words suitable forthe mood of the music audio. We conclude thatembedding multiple aspects into a vector spacecontributes to capturing atypicality and relationshipwith audio.

Acknowledgements. The authors appreciate
SyncPower Corporation for providing lyrics data.
This work was supported in part by JST ACCEL Grant Number JPMJAC1602, JST CREST
Grant Number JPMJCR20D4, and JSPS KAKENHI Grant Number 20K19878, Japan.

References
Gabriele Barbieri, François Pachet, Pierre Roy, andMirko Degli Esposti. 2012. Markov constraints forgenerating lyrics with style. In Proceedings of the20th European Conference on Artificial Intelligence,
pages 115–120.
Sepp Hochreiter and Jürgen Schmidhuber. 1997.
Long short-term memory. Neural computation,
9(8):1735–1780.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In Proceedingsof the 3rd International Conference on Learning
Representations (ICLR 2015).
Yang Liu, Wan-Lei Zhao, Chong-Wah Ngo, ChangSheng Xu, and Han-Qing Lu. 2010. Coherent bagof audio words model for efficient large-scale video
copy detection. In Proceedings of the ACM International Conference on Image and Video Retrieval,pages 89–96.
Oren Melamud, Jacob Goldberger, and Ido Dagan.
2016. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings ofthe 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th Annual Conference
on Neural Information Processing Systems, pages
3111–3119.
Hugo R. Gonçalo Oliveira, F. Amialcar Cardoso, andFrancisco C. Pereira. 2007. Tra-la-lyrics: an approach to generate text based on rhythm. In Proceedings of the 4th International Joint Workshop onComputational Creativity, pages 47–55.
Peter Potash, Alexey Romanov, and Anna Rumshisky.
2015. GhostWriter: Using an LSTM for automatic
Rap lyric generation. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language Processing, pages 1919–1924.
Kento Watanabe and Masataka Goto. 2019. Query-byBlending: A music exploration system blending latent vector representations of lyric word, song audio,and artist. In Proceedings of the 20th annual conference of the International Society for Music Information Retrieval, pages 144–151.

Kento Watanabe and Masataka Goto. 2021. Atypicallyrics completion considering musical audio signals.
In Proceedings of the 27th International Conference
on Multimedia Modeling, volume 12572, pages 174–186.
Kento Watanabe, Yuichiroh Matsubayashi, Satoru
Fukayama, Masataka Goto, Kentaro Inui, and Tomoyasu Nakano. 2018.
A melody-conditionedlyrics language model. In Proceedings of the 16th
Annual Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 163–172.
Kento Watanabe, Yuichiroh Matsubayashi, KentaroInui, Tomoyasu Nakano, Satoru Fukayama, andMasataka Goto. 2017. LyriSys: An interactive support system for writing lyrics based on topic transition. In Proceedings of the 22nd Annual Meetingof the Intelligent User Interfaces Community, page559—563.