IEICE TRANS. INF. & SYST., VOL.E106–D, NO.9 SEPTEMBER 2023

1600

PAPER

A Method to Detect Chorus Sections in Lyrics Text∗
Kento WATANABE†a) , Nonmember and Masataka GOTO†b) , Fellow

SUMMARY
This paper addresses the novel task of detecting chorus
sections in English and Japanese lyrics text. Although chorus-section detection using audio signals has been studied, whether chorus sections canbe detected from text-only lyrics is an open issue. Another open issue iswhether patterns of repeating lyric lines such as those appearing in chorus sections depend on language. To investigate these issues, we propose
a neural-network-based model for sequence labeling. It can learn phrase
repetition and linguistic features to detect chorus sections in lyrics text. It
is, however, diﬃcult to train this model since there was no dataset of lyrics
with chorus-section annotations as there was no prior work on this task. We
therefore generate a large amount of training data with such annotations byleveraging pairs of musical audio signals and their corresponding manually
time-aligned lyrics; we first automatically detect chorus sections from theaudio signals and then use their temporal positions to transfer them to theline-level chorus-section annotations for the lyrics. Experimental results
show that the proposed model with the generated data contributes to detecting the chorus sections, that the model trained on Japanese lyrics candetect chorus sections surprisingly well in English lyrics, and that patternsof repeating lyric lines are language-independent.
key words: lyrics information processing, music information retrieval, natural language processing, lyrics structure analysis

1.

Introduction

The digitization of lyrics collections has opened various areas of lyrics-based research, such as research on
lyrics browsing [2]–[4], lyrics genre classification [5]–[7]and lyrics-to-audio synchronization [8]–[18]. Lyrics areusually plain text without any annotations, and some researchers have analyzed their structure, such as paragraphstructure and topic transitions between paragraphs [19]–[23]. For example, Fell et al. [19] and Watanabe et al. [20]estimated section boundaries in lyrics text without empty
lines but were not able to assign a section label such as verseor chorus to each estimated section. Chorus sections were
not detected in lyrics text.
The goal of this paper is to achieve automatic chorussection detection for lyrics text. This task has not been studied, though chorus-section detection, as well as music structure analysis, for audio signals has been a popular topic ofresearch [24]–[42]. Since whether chorus sections can be
Manuscript received July 29, 2022.
Manuscript revised March 30, 2023.
Manuscript publicized June 2, 2023.
†
The authors are with National Institute of Advanced Industrial
Science and Technology (AIST), Tsukuba-shi, 305–8568 Japan.
∗
An earlier version of this paper was published at a conference [1].
a) E-mail: kento.watanabe@aist.go.jpb) E-mail: m.goto@aist.go.jp
DOI: 10.1587/transinf.2022EDP7139

detected from text-only lyrics is an open issue, it is worth
investigating this issue from an academic viewpoint. Moreover, a chorus-section detection method for lyrics text has
potential applications. For example, when listeners want tofind lyrics with a chorus section having a particular phrase
such as “I love you” for the purpose of singing that section or reusing it in a short video clip, it is necessary for alyrics search system to automatically detect which lines ofthe lyrics are included in chorus sections. The detected lyric
lines of chorus sections could be used in a lyrics viewing
function of music services displaying lyrics with those lines
highlighted by a diﬀerent color or typeface. Automatic lyric
video generation technologies could give those lines more
vivid animations.
Chorus sections are the most repeated and memorable
portions of a song [40]. Since it is not easy to find suchsections by exploiting heuristic rules, most existing chorussection detection methods for audio signals have leveraged
repetitive patterns of those sections within a song. In thispaper, we propose a supervised model that can detect chorus sections in English and Japanese lyrics. Our model
uses both structural features that represent patterns of repeating lyric lines, and linguistic features that are calculated either from word2vec [43] with context2vec [44] orfrom BERT [45]. To detect chorus sections using only plain
text without any labels or even empty lines (i.e., section
boundaries), we investigate a model and features eﬀective
for chorus-section detection. Experimental results show thatour proposed model outperforms alternative baseline models and that combining structural and linguistic features contributes to better performance.
Although such a supervised model needs a large datasetof lyrics with line-level chorus-section annotations for its
training, there was no such dataset as there was no priorwork on chorus sections in lyrics text. To address this issue of lacking training data, we generated a dataset consisting of 9,313 English and 91,459 Japanese lyrics withchorus-section annotations by utilizing pairs of musical audio signals and their corresponding manually time-aligned
lyrics. We first automatically detected chorus sections inaudio signals of a song [40]. Then, since each lyric line
had the corresponding start time within the song, we could
find lyric lines that temporally correspond to the durationof each detected chorus section. We thus obtained the annotated dataset by assigning a chorus label to those lyric
lines and a not-chorus label to the other lines. Experimental results show that the model trained with this large auto-c 2023 The Institute of Electronics, Information and Communication Engineers
Copyright WATANABE and GOTO: A METHOD TO DETECT CHORUS SECTIONS IN LYRICS TEXT

1601

Fig. 1 Example of lyrics with chorus-section annotations and corresponding self-similarity matrix inwhich each cell represents the similarity between two lyric lines. These lyrics are from “How Deep IsYour Love?” (RWC-MDB-P-2001 No. 81 in the RWC Music Database [46]).

matically generated dataset performs better than the modeltrained with a smaller manually annotated dataset and thatthe model trained on Japanese lyrics can detect chorus sections surprisingly well in English lyrics.
2.

Lyrics Chorus-Section Detection Task

The left side of Fig. 1 shows an example of lyrics withchorus-section annotations (labels). The lyrics of a song
are a sequence of lyric lines, each line having a sentenceor phrase. In this example there are three highlighted chorus sections that have exactly the same four lines, though inother songs, lyrics of chorus sections are repeated with somemodifications. To maximize the applicability, as shown inthis example, we assume that the input text of lyrics does nothave any section boundaries. Even though some lyrics contain empty lines at those boundaries, those lines are deleted
in advance. We also assume that the input text does not have
explicit chorus labels such as “(chorus)” at the beginningsof chorus sections. Even though some lyrics contain thoselabels, they are deleted as well. When lyrics contain a repetition label such as “(* repeat)”, it is manually replaced withthe corresponding lyric lines.
We formulate this chorus-section detection task asa sequence labeling problem: predicting the chorus or
not-chorus status for each lyric line. Let X s be thelyrics of a song s composed of T lines of text: X s ={x1 , . . . , xt , . . . , xT }. Each lyric line xt has a binary label yt .
If yt = 1, xt is in a chorus section. If yt = 0, xt is not in achorus section. Y s denotes a sequence of labels correspond-

ing to X s : Y s = {y1 , . . . , yt , . . . , yT }. In the training step,
the model learns the conditional probability P(Y s |X s ). In thevalidation/testing step, the trained model has to predict labels Y s for given lyric lines X s .
Chorus sections cannot be detected by simply extracting repeated lines since those lines often correspond to nonchorus sections. For example, lyric lines 9–12 and 21–24
in Fig. 1 are exactly repeated, but those lines are not in chorus sections. It is also diﬃcult to manually define a set ofrules to find various chorus sections. We therefore prepare
various features that could be useful for machine learning todeal with various types of chorus sections.
3.

Computational Modeling of Chorus Sections in
LyricsWe propose a neural-network-based model for sequence labeling by using structural features that are self-similarity
matrix (SSM) representations. SSM representations arewidely used in computational music structure analysis, but
we use diﬀerent representations for lyrics. In addition to
structural features, our model utilizes two kinds of linguistic features widely used in natural language processing (NLP): (1) word vectors and sentence vectors calculated
from word2vec [43] and context2vec [44], and (2) sentence
vectors calculated from BERT [45].
In the following sections, we first describe nine SSMs
for capturing patterns of repeating lyric lines and explain
how to encode the SSMs for neural networks (Sect. 3.1). We
then describe the linguistic features obtained by vectorizing

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.9 SEPTEMBER 2023

1602

the semantic/syntactic information of lines using word2vec,
context2vec, and BERT (Sect. 3.2). Finally, we describe aneural-network-based sequence labeling model with thesestructural and linguistic features (Sect. 3.3).
3.1

Structural Features

Most previous work on music structure analysis for audio
signals [24]–[42] identifies repeated musical sections by using a SSM like that shown in Fig. 1. Repeated sections leadto high values in diagonals of the matrix, and those patterns
are used to identify the structure. To capture repeated lyric
lines that often appear in chorus sections, we also compute
the SSM from lyrics text, but the design of the similarity
measure to compute each cell of the SSM is important. We
propose to use the following nine variations of similarity
measures simm , where m denotes the variation. Some of thesimilarities are based on previous studies [19], [20].
1. String similarity (sim str ): a normalized Levenshtein
edit distance [47] between the characters of two lyric
lines.
2. Head similarity (simhead ): a normalized Levenshtein
edit distance between the characters of the first twowords of two lyric lines.
3. Tail similarity (simtail ): a normalized Levenshtein edit
distance between the characters of the last two wordsof two lyric lines.
4. Phonetic similarity (sim phone ): To capture rhymes in
the lyrics, we calculate a normalized Levenshtein edit
distance between the phonetic transcriptions of two
lyric lines. We use the CMU pronunciation dictionary†
to extract the phonetic transcription. For example, thephonetic transcription of “I love you” is [AY1, L, AH1,
V, Y, UW1].
5. Part-of-speech similarity (sim pos ): To capture similarities in grammatical structure, we calculate a normalized Levenshtein edit distance between the part-ofspeech (POS) sequences of two lines. We use the default POS tagger in the NLTK package [48].
6. Word vector similarity (simw2v ): To capture the semantic similarity between two lyric lines, we simply
average vectors of the words of each lyric line by using pre-trained word2vec [43] and compute their cosine
similarity. This “bag of words” representation does
not diﬀerentiate “dog bites person” from “person bitesdog”.
7. Context vector similarity (simc2v ): To consider theword order, we vectorize the lyric lines using pretrained context2vec [44], an extension of word2vec,
which encodes a sequence of words by using Long
Short-Term Memory (LSTM) networks [49]. We thencompute their cosine similarity to obtain simc2v .
8. Word syllable count similarity (sim syW ): Since repeated phrases sometimes have the same number ofsyllables even if their words are diﬀerent, we use a†

http://www.speech.cs.cmu.edu/cgi-bin/cmudict

Fig. 2

Convolutional neural network for SSMs.

sequence of word syllable counts on each lyric line.
For example, the word syllable counts of the two lyric
lines “Sometimes you lost yourself away” and “Everytime you just close your eyes”†† are {2, 1, 1, 2, 1} and{2, 1, 1, 1, 1, 1}, respectively. When successive lyric
lines have similar syllable count sequences, they arelikely to correspond to the repetition of sections. We
use dynamic time warping (DTW) [50] to calculate thesimilarity between syllable count sequences.
9. Lyric Line syllable count similarity (sim syL ): We
can also use the total syllable count of all words ineach lyric line. For example, in all the chorus sections shown in Fig. 1, the total syllable count of thefirst lyric line is 6 and that of the second line is 8.
We calculate the similarity of such total syllable countsof each pair of lyric lines by using the following procedure. (1) We extract a window of four lyric lines
Lt = {xt , xt+1 , xt+2 , xt+3 } and shift it over the entire
lyrics of a song. (2) The similarity between the lyric
lines xt and xt is calculated by DTW of Lt and Lt .
We thus calculated nine SSMs Am ∈ RT ×T , where eachcell is a simm explained above. Then, to calculate feature
vectors from the above nine SSMs, we exploit a convolutional neural network (CNN) architecture to detect textual
macro structures from various patterns in SSMs regardlessof their locations and relative sizes in SSMs. Except for network parameters, this CNN architecture is the same as thatof Fell et al. [19], as we share the same motivation: to extract translation, scaling and rotation invariant features fromthe input image (in our case, nine SSMs).
Figure 2 illustrates the CNN structure. After calculating the nine SSMs, we extract fixed-size elongated-rectangle††
These lyrics are taken from the RWC Music Database (RWCMDB-P-2001 No. 92) [46].

WATANABE and GOTO: A METHOD TO DETECT CHORUS SECTIONS IN LYRICS TEXT

1603

sub-matrices centered on the target lyric line xt : atm =Am [t − w + 1, . . . , t + w; 1, . . . , T ] ∈ R2w×T , where w is a fixedwindow size. The input of the CNN is nine sub-matrices{atstr , . . . , atsyL } ∈ R2w×T ×9 , where the number of channels
corresponds to the number of SSMs (i.e., 9). The kernel
size of the first 2D-convolutional layer is (w + 1) × (w + 1)
so that each feature can capture a prospective chorus section. This first 2D-convolutional layer has 200 kernels, so
this convolution produces 200 matrices from the nine submatrices (bottom left of Fig. 2). Each of the resulting 200
matrices is downsampled to a vector by max-pooling withw × w kernel size, so this produces 200 vectors (bottom center of Fig. 2). We then apply the 1D-convolutional layer witha kernel size of w to the 200 vectors (200 channels). This
1D-convolutional layer has 400 kernels, so the convolution
produces 400 vectors from the 200 vectors (bottom right ofFig. 2). The final max-pooling layer reduces the dimensionof each of the resulting 400 vectors to one dimension, resulting in the final 400-dimensional vector ut .
In this network, all convolutional layers employ the
ReLU function. We perform the above procedure independently for each lyric line xt and obtain the CNN-based feature vector ut . We call this vector ut the structural featuresimall .
3.2

Linguistic Features

Some expressions tend to appear in chorus sections. To
quantify this tendency, we calculate the diﬀerence between
word tri-gram probabilities in the chorus and non-chorus
sections. Table 1 shows the word tri-grams that frequently
appear in both of the sections. Here, Pc and Pn denote word
tri-gram probabilities in the chorus and non-chorus sections,
respectively. As shown in this table, we found that phrases
about the future (e.g., “I’ll” and “Let’s”) tend to appear inchorus sections more often than do phrases about the past(e.g., “have been” and “didn’t”). To exploit such tendencies, we compute two kinds of linguistic features:words. Since the word order cannot be modeled byword2vec, we then use pre-trained context2vec [44]that puts a sequence of the word vectors based on
word2vec into the LSTM to obtain a sentence vector.
We finally concatenate the averaged word2vec-based
word vector with the context2vec-based sentence vector to obtain the concatenated vector utave+seq . We callthis vector utave+seq the linguistic feature lingave+seq .
2. BERT-based linguistic feature (lingBERT ): Since
BERT has been reported to improve performance invarious NLP tasks [45], we calculate the BERT-based
feature vector utBERT in addition to utave+seq so that we
can compare them. We first feed each lyric line xt to thepre-trained BERT model. Among the output vectorsof BERT, we then obtain the vector resulting from theposition of the [CLS] token and use it as the sentence
embedding vector utBERT . We call this vector utBERT the
linguistic feature lingBERT .
As lingBERT is expected to be better than lingave+seq , we compared their performances in our experiments.
3.3 Neural-Network-based Sequence Labeling Model
To solve the sequence labeling problem, we use the standard
Bidirectional Long Short-Term Memory (Bi-LSTM) networks [51] to compute the conditional probability P(Y s |X s ).
The neural network structure is illustrated in Fig. 3.
The input to the Bi-LSTM layer at each time step t(lyric line xt ) is a concatenation of two diﬀerent types offeature vectors: (1) the structural feature vector ut encodedfrom the nine variations of SSMs in Sect. 3.1 and (2) the linguistic feature vector utave+seq or utBERT encoded in Sect. 3.2.
Formally, the conditional probability P(Y s |X s ) is calculated
by using a softmax function:

1. word2vec/context2vec-based linguistic feature(lingave+seq ): For each lyric line xt , we first calculate the average of word vectors obtained using
pre-trained word2vec [43], skipping out-of-vocabulary
Table 1 Frequent word tri-grams in chorus and non-chorus sections. An
apostrophe is regarded as a word.
Tri-gramI’m
don’t
oh oh oh
I’ll
we’re
you’re’ll beI don’
Let’s
you got ta
I can’
can’t

Pc − Pn
0.12%
0.11%
0.05%
0.05%
0.04%
0.04%
0.04%
0.04%
0.03%
0.03%
0.03%
0.03%

Tri-gram
there’s
I’ve
’s a
I’d
but I’’s notwhat’sna na nayeah yeah yeah’ve been’t takedidn’tPn − Pc
0.04%
0.03%
0.03%
0.02%
0.02%
0.01%
0.01%
0.01%
0.01%
0.01%
0.01%
0.01%

Fig. 3 Neural-network-based sequence labeling model for chorussection detection. The BERT-based feature vector utBERT is used as the
linguistic feature in this figure.

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.9 SEPTEMBER 2023

1604

P(Y s |X s ) = 

exp(Score(X s , Y s )).
Y s exp(Score(X s , Y s ))The Score() is defined as
Score(X s , Y s ) =BN(ht [yt ]),

(1)

(2)

t

where ht [yt ] is the output of the Bi-LSTM for each time stept and BN() denotes batch normalization [52]. In the modeltraining step, we use a binary cross-entropy loss.
4.

Experiment

Inspired by audio-based chorus-section detection [40], we
evaluated the proposed method by using the F-measure (F)that is a harmonic mean of precision (P) and recall (R), F =(2 · R · P)/(R + P), where# of lyric lines in correctly detected chorus sections.
P=# of lyric lines in detected chorus sections# of lyric lines in correctly detected chorus sections
R = # of lyric lines in correct (annotated) chorus sections .

We also used the pair-wise F-measure (p-F), normalized
conditional entropy F-measure (n-F) and V-measure (V)that are provided by the Python module mir eval and commonly used to evaluate computational music structure analysis [53].
4.1

Methods Compared

To confirm the eﬀectiveness of our Bi-LSTM method basedon the Bi-LSTM model that can learn dependencies between
adjacent lyric lines, we compared its performance with thoseof two baseline methods:
1. Heuristic: We implemented the heuristic that “if lines
at the end of the lyrics are repeated with small modifications, all those repeated lines are chorus sections”
by the following procedure: (i) From the SSM thatis the average of the nine SSMs, we extracted diagonals whose cells had values higher than a threshold λ,which was tuned on a development set to be 0.62. (ii)
From the extracted diagonals, we selected the shortest
diagonal among diagonals placed at the bottom of the
SSM (e.g., the diagonal starting at the cell SSM[29; 1]
in Fig. 1). (iii) Successive lines corresponding to therows where the selected diagonal was located (e.g.,
lyric lines 29–32 in Fig. 1) were assigned the label
chorus. (iv) Other successive lines that were similar
to the chorus lines (e.g., lyric lines 1–4 and 13–16 in
Fig. 1) were also assigned chorus labels.
2. Multi-Layer Perceptron (MLP): Similar to the BiLSTM method, but with the Bi-LSTM model replaced
by a standard MLP model. This method ignores transitions between adjacent lyric lines and predicts yt fromxt only.
We chose the number of kernels for the first and second CNNs to be 200 and 400, respectively. We used w = 3

for the window size. In the MLP and Bi-LSTM methods,
we chose the dimension of the hidden state to be 600.
The word2vec [43] and context2vec [44] were pre-trainedon lyrics and were not updated in the model training stepof our method. The dimension of their output vectors was
300. We used pre-trained BERT models [45] that are publicly available† . The dimension of the BERT-based feature
vector is 768. We used AdamW for parameter optimization [54]. The initial learning rate was 0.001 with an exponential decay. We used a mini-batch size of 64. Training
was run for 100 epochs, and the model used for testing was
the one that achieved the best F-measure on the development
set.
4.2 Dataset
To train our computational model that predicts whether thelabel of each lyric line is chorus or not-chorus, we
needed a large amount of lyrics data with line-level chorussection annotations like those illustrated in Fig. 1. Since
there was no dataset for this, we generated a large amountof such lyrics data by the following procedure:
1. We prepared 100,772 pairs of musical audio signals
and their corresponding manually time-aligned (temporally synchronized) lyrics†† . To avoid unreliable
lyrics, we made sure that all lyrics had more than eightlines and less than 120 lines.
2. We detected chorus sections of every song automatically by using its audio signals. In our experiments, weused the RefraiD method [40] to obtain the start andend times of each chorus section, but other methods
could also be used.
3. If the start time of a lyric line was within any chorus
section detected in audio signals, that line was labeledchorus; otherwise, it was labeled not-chorus.
Of course, not all generated annotations were correct, but
by using over 100,000 training data, the model could be robustly trained without being influenced by errors or outliers.
The generated training data consisted of 9,313 English and91,459 Japanese songs, and we called them EN auto and
JA auto, respectively.†††Furthermore, we manually annotated three sets oflyrics data with more reliable line-level chorus-section annotations for three diﬀerent purposes:(a) For training comparison:

We annotated 1,103†
We used https://huggingface.co/bert-base-uncased for English and https://huggingface.co/cl-tohoku/bert-base-japanese-v2
for Japanese.
††In our experiments, English and Japanese lyrics text as well as
the start time of every lyric line were provided by a lyrics distribution company. Automatic lyrics-to-audio synchronization [8]–[18]could also be used to estimate such start times.
†††
The main genres are Rock (33%), Pop (25%), and Alternative(12%) for EN auto and are J-Pop (53%), Rock (20%), and Anime(9%) for JA auto.

WATANABE and GOTO: A METHOD TO DETECT CHORUS SECTIONS IN LYRICS TEXT

1605
Table 3

Experimental result: Importance of using both structural and linguistic features.

Featuresimalllingave+seq
lingBERT
simall + lingave+seq
simall + lingBERTTable 2is %).

MethodHeuristic
MLP
Bi-LSTM

Training data / Testing data
EN auto / EN test
JA auto / JA test
F
p-F
n-F
V
F
p-F
n-F
V
77.9 76.1 48.6 45.5
81.2 82.7 63.6 59.6
57.4 59.9 16.5
6.9
55.2 61.8 22.1 16.7
61.7 61.1 19.9 12.7
58.3 66.6 29.3 23.3
78.1 77.7 50.8 47.3
83.4 83.5 64.9 61.4
79.7 78.2 51.7 47.8
85.4 83.3 65.3 62.7

Experimental result: Comparison of diﬀerent methods (the unit
Training data / Testing data
EN auto / EN test
JA auto / JA test
F
p-F
n-F
V
F
p-F
n-F
V
57.8 73.8 43.0 35.8 57.1 73.2 43.6 36.3
76.4 72.7 43.6 39.4 79.8 83.0 62.3 58.9
79.7 78.2 51.7 47.8 85.4 83.3 65.3 62.7

Japanese lyrics and called them JA man† . By comparing the performance of the model trained on JA autowith that of the model trained on JA man, we could
confirm that our generated data is reliable enough fortraining purposes.
(b) For tuning model parameters: We annotated the
lyrics of 21 English and 79 Japanese songs from RWCMDB-P-2001 and called them EN RWC and JA RWC,
respectively. These were used to tune model parameters.
(c) For testing: We annotated the lyrics of 118 other English songs and 128 other Japanese songs and called
them EN test and JA test, respectively†† . These wereused to test the chorus-section detection methods.
4.3

Comparison of Diﬀerent Methods

Table 2 summarizes the evaluated performances of Heuristic, MLP, and the proposed Bi-LSTM. Note that in this section, MLP and Bi-LSTM are trained using the structural feature simall and the linguistic feature lingBERT . We found that
MLP and Bi-LSTM performed better than Heuristic. This
indicates that methods based on supervised learning are better than a rule-based method. We also found that Bi-LSTM
was better than MLP and thus confirmed the importance oflearning dependencies between adjacent lines.
Since we concluded from these results that the proposed Bi-LSTM is the best for the chorus-section detectiontask, in the subsequent experiments reported here we used
only Bi-LSTM.
†
To investigate the accuracy of the automatic annotation
method we used for generating EN auto and JA auto, we applied
the same method to the songs (audio signals and corresponding
manually time-aligned lyrics) in JA man. The accuracy of the generated annotations was F = 68.0%, so the automatic annotation
method seems to work fairly well.
††
The chorus and not-chorus labels were annotated only on
the lyrics. No audio signal is available for these test data.

4.4 Importance of Using Both Structural and Linguistic
FeaturesTo investigate the eﬀectiveness of structural and linguisticfeatures, we compared their use individually and in combination. Table 3 summarizes the results.
The top three entries in Table 3 show that the models trained solely on the structural feature simall greatly outperformed the models trained solely on the linguistic feature lingave+seq or lingBERT . This result confirms that capturing the repetitive structure is eﬀective in detecting chorus sections not only in audio signals but also in lyrics
text. We also confirmed that the use of the BERT-based
linguistic feature lingBERT outperformed the use of theword2vec/context2vec-based linguistic feature lingave+seq ,as expected.
The bottom two entries in Table 3 show that combining the structural feature simall with the linguistic feature
lingave+seq or lingBERT improved performance. This not only
confirms the importance of using SSMs, as had been shownfor the audio-based detection of chorus sections, but also
confirms that the additional use of linguistic features is helpful for detecting chorus sections, which has not been shown
before.
Table 3 also shows that the combination of simall +lingBERT outperformed the combination of simall +lingave+seq , as expected. However, the performance diﬀerence between those combinations (the bottom two entries)
was much smaller than the performance diﬀerence between
lingBERT and lingave+seq (the second and third entries fromthe top). This confirms that although BERT is superior toword2vec/doc2vec as a linguistic feature, the combinationof structural and linguistic features reduces its superiority
since the structural feature is much more eﬀective for the
lyric chorus-section detection task.
Since we concluded from these results that lingBERT issuperior to lingave+seq whether it is used alone or in combination, in the subsequent experiments reported here we used
only lingBERT .
4.5 Reliability of Generated Annotations
To investigate whether JA auto containing some annotation
errors is reliable enough for training purposes, we compared
the model trained using JA auto with the model trained using JA man that does not contain annotation errors. Table 4

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.9 SEPTEMBER 2023

1606
Table 4 Experimental result: Reliability of automatically generated annotations.
Training data
F
p-F
n-F
V
JA auto (91,459 songs) 85.4 83.3 65.3 62.7
81.1 77.8 54.7 52.0
JA man (1,103 songs)

Table 5 Experimental result: Can the Japanese model detect English
chorus sections?
Training data
Testing data
F
p-F
n-F
V
EN auto (9,313 songs)
EN test77.9 76.1 48.6 45.5
EN test80.3 80.6 58.1 54.4
JA auto (91,459 songs)
EJ auto (100,772 songs)
EN test81.0 82.3 60.7 57.4

trained on both languages performed better than the modeltrained on only one.
These results confirm that chorus sections can be detected by a model trained on data in another language, thatpatterns of repeating lyric lines are language-independentand that mixing diﬀerent language data allows the model tolearn the general structure of chorus sections and thereby
perform better. This could have an impact on low-resource
languages because large-scale training data can be createdby mixing other available language resources.
5.

clearly shows that the model trained using JA auto, automatically generated data the amount of which can be large,
outperformed the model trained using JA man, manually annotated data, the amount of which is usually very limited
because of the laborious manual eﬀort its creation requires.
The result also confirms that even if annotations generated
automatically are not perfect, they are reliable enough fortraining the model.
4.6

Training Data Size and Language Dependency

Tables 2 and 3 also show that the performances for English
lyrics were worse than those for Japanese lyrics. Since theamount of Japanese training data was about 10 times that ofEnglish training data, we think that the amount of training
data greatly aﬀects the performance of the proposed model.
We are thus interested in answering the question “Can amodel trained on a large amount of Japanese data detect
English chorus sections?” In fact, although linguistic features are language dependent and the process of computing
SSMs is also language dependent, structural features basedon the resulting SSMs can be language independent because
our SSMs simply represent patterns of repeating lyric lines,
which could be universal in music. Therefore, in this section, we compare models solely trained on the structural features, without using the linguistic features.
As shown in the upper half of Table 5, which shows
results obtained without using linguistic features, we foundthat the structural-feature-based model trained on Japanesedata JA auto succeeded in detecting English chorus sections
in EN test and its performance was better than that of themodel trained on the smaller dataset EN auto. This result indicates that the SSM-based model trained on a large amountof data can detect chorus sections regardless of the languageof the test set. Moreover, this result is further evidence thatJapanese and English SSMs (i.e., patterns of repeating lyric
lines) have similar structures.
Obviously, the above result raises another question:“Can a model trained on both EN auto and JA auto perform better than one trained on only EN auto or JA auto?”
To answer this question, we created training data EJ autoby including both EN auto and JA auto and constructed yet
another structural-feature-based model with EJ auto. As
shown in the lower half of Table 5, we found that the model

Related Work

Previous work in the community of music information retrieval has addressed musical structure analysis and chorussection detection based on repeated patterns in musical audio signals [24]–[42]. Studies in the chorus-section detection for audio signals typically used SSMs to capture repeated structures, and we share this motivation. Our approach diﬀers from those audio-based approaches in that it
exploits multiple lyrics-based SSMs and linguistic features
within chorus sections.
On the other hand, recent work in the community ofnatural language processing has tackled lyrics segmentationand summarization tasks by exploiting SSMs. Fell et al. [19]and Watanabe et al. [20] proposed a neural network modeland logistic regression model for segmenting paragraphs(sections) without labeling them by using SSMs as features.
Those tasks, however, are essentially diﬀerent from detecting all chorus sections that are the most representative sections in lyrics text. Addressing a task similar to chorussection detection, Fell et al. [55] proposed a method of summarizing lyrics by combining general document summarization methods with audio thumbnailing methods. They focus on extracting individual informative lines as a summaryfrom lyrics text, not redundant repeated lines. On the other
hand, the focus of our paper is to detect chorus sections
whose successive lines are often repeated in lyrics text.
6.

Conclusion

This paper has addressed the novel task of detecting chorus sections in English and Japanese lyrics. We proposeda neural-network-based sequence labeling model that learns
structural (i.e., phrase-repetition) and linguistic features todetect lyric lines of chorus sections. We also generated over
100,000 training data with chorus-section annotations. No
previous work has ever conducted chorus-section detection
for text-only lyrics with this much data.
The contributions of this study are summarized as follows: (1) We designed a variety of features to capture structural and linguistic properties of chorus sections. (2) We
proposed a sequence labeling model that can detect chorus
sections in lyrics. (3) We showed how to generate a large
training dataset of lyrics with chorus-section annotations.
(4) We demonstrated that our Bi-LSTM-based method outperforms alternative baseline methods. (5) We thoroughlyWATANABE and GOTO: A METHOD TO DETECT CHORUS SECTIONS IN LYRICS TEXT

1607

investigated this detection task and the nature of chorus sections of lyrics from diﬀerent perspectives such as the importance of features, the amount of training data, and language
dependency.
We plan to extend our method to detect other sections,such as verse and bridge sections. Future work will also
develop music applications using our method, such as those
discussed in Sect. 1.
Acknowledgments
The authors appreciate SyncPower Corporation for providing lyrics data. This work was supported in part by JST
ACCEL Grant Number JPMJAC1602, JST CREST Grant
Number JPMJCR20D4, and JSPS KAKENHI Grant Number JP20K19878.
References[1] K. Watanabe and M. Goto, “A chorus-section detection method for
lyrics text,” Proceedings of the 21st International Society for Music
Information Retrieval Conference (ISMIR 2020), pp.351–359, 2020.
[2] K. Watanabe and M. Goto, “Query-by-Blending: A music exploration system blending latent vector representations of lyric word,
song audio, and artist,” Proceedings of the 20th International Society
for Music Information Retrieval Conference (ISMIR 2019), pp.144–
151, 2019.
[3] K. Tsukuda, K. Ishida, and M. Goto, “Lyric Jumper: A lyrics-based
music exploratory web service by modeling lyrics generative process,” Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR 2017), pp.544–551, 2017.
[4] S. Sasaki, K. Yoshii, T. Nakano, M. Goto, and S. Morishima,“LyricsRadar: A lyrics retrieval system based on latent topics oflyrics,” Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014), pp.585–590, 2014.
[5] A. Tsaptsinos, “Lyrics-based music genre classification using a hierarchical attention network,” Proceedings of the 18th International
Society for Music Information Retrieval Conference (ISMIR 2017),
pp.694–701, 2017.
[6] R. Mayer and A. Rauber, “Music genre classification by ensemblesof audio and lyrics features,” Proceedings of the 12th International
Society for Music Information Retrieval Conference (ISMIR 2011),
pp.675–680, 2011.
[7] R. Mayer, R. Neumayer, and A. Rauber, “Rhyme and style features
for musical genre classification by song lyrics,” Proceedings of the
9th International Conference on Music Information Retrieval (ISMIR 2008), pp.337–342, 2008.
[8] B. Sharma, C. Gupta, H. Li, and Y. Wang, “Automatic lyrics-to-audio alignment on polyphonic music using singing-adapted acoustic
models,” Proceedings of the 2019 IEEE International Conferenceon Acoustics, Speech and Signal Processing (IEEE ICASSP 2019),
pp.396–400, 2019.
[9] D. Stoller, S. Durand, and S. Ewert, “End-to-end lyrics alignment for polyphonic music using an audio-to-character recognition
model,” Proceedings of the 2019 IEEE International Conferenceon Acoustics, Speech and Signal Processing (IEEE ICASSP 2019),
pp.181–185, 2019.
[10] C. Gupta, E. Yılmaz, and H. Li, “Acoustic modeling for automatic
lyrics-to-audio alignment,” Proceedings of the 20th Annual Conference of the International Speech Communication Association (Interspeech 2019), pp.2040–2044, 2019.
[11] C. Gupta, R. Tong, H. Li, and Y. Wang, “Semi-supervised lyrics
and solo-singing alignment,” Proceedings of the 19th International
Society for Music Information Retrieval Conference (ISMIR 2018),

pp.600–607, 2018.
[12] S. Chang and K. Lee, “Lyrics-to-audio alignment by unsupervised
discovery of repetitive patterns in vowel acoustics,” IEEE Access,vol.5, pp.16635–16648, 2017.
[13] S.W. Lee and J. Scott, “Word level lyrics-audio synchronization using separated vocals,” Proceedings of the 2017 IEEE International
Conference on Acoustics, Speech and Signal Processing (IEEEICASSP 2017), pp.646–650, 2017.
[14] Y.-R. Chien, H.-M. Wang, and S.-K. Jeng, “Alignment of lyrics withaccompanied singing audio based on acoustic-phonetic vowel likelihood modeling,” IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol.24, no.11, pp.1998–2008, 2016.
[15] G.B. Dzhambazov and X. Serra, “Modeling of phoneme durations
for alignment between polyphonic audio and lyrics,” Proceedingsof the 12th Sound and Music Computing Conference (SMC 2015),
pp.281–286, 2015.
[16] M. Mauch, H. Fujihara, and M. Goto, “Integrating additional chord
information into HMM-based lyrics-to-audio alignment,” IEEE
Transactions on Audio, Speech, and Language Processing, vol.20,
no.1, pp.200–210, 2012.
[17] H. Fujihara, M. Goto, J. Ogata, and H.G. Okuno, “LyricSynchronizer: Automatic synchronization system between musical audio
signals and lyrics,” IEEE Journal of Selected Topics in Signal Processing, vol.5, no.6, pp.1252–1261, 2011.
[18] M.-Y. Kan, Y. Wang, D. Iskandar, T.L. Nwe, and A. Shenoy,“LyricAlly: Automatic synchronization of textual lyrics to acoustic
music signals,” IEEE Trans. Speech Audio Process., vol.16, no.2,
pp.338–349, 2008.
[19] M. Fell, Y. Nechaev, E. Cabrio, and F. Gandon, “Lyrics segmentation: Textual macrostructure detection using convolutions,” Proceedings of the 27th International Conference on Computational
Linguistics (COLING 2018), pp.2044–2054, 2018.
[20] K. Watanabe, Y. Matsubayashi, N. Orita, N. Okazaki, K. Inui,
S. Fukayama, T. Nakano, J.B.L. Smith, and M. Goto, “Modeling
discourse segments in lyrics using repeated patterns,” Proceedingsof the 26th International Conference on Computational Linguistics(COLING 2016), pp.1959–1969, 2016.
[21] A. Baratè, L.A. Ludovico, and E. Santucci, “A semantics-driven approach to lyrics segmentation,” Proceedings of the 8th International
Workshop on Semantic and Social Media Adaptation and Personalization (SMAP 2013), pp.73–79, 2013.
[22] J.P.G. Mahedero, Á. Martı́nez, P. Cano, M. Koppenberger, and F.
Gouyon, “Natural language processing of lyrics,” Proceedings of the
13th ACM International Conference on Multimedia (ACM Multimedia), pp.475–478, 2005.
[23] K. Watanabe, Y. Matsubayashi, K. Inui, S. Fukayama, T. Nakano,and M. Goto, “Modeling storylines in lyrics,” IEICE Transactions on
Information and Systems, vol.E101.D, no.4, pp.1167–1179, 2018.
[24] G. Shibata, R. Nishikimi, E. Nakamura, and K. Yoshii, “Statisticalmusic structure analysis based on a homogeneity-, repetitiveness-,and regularity-aware hierarchical hidden semi-markov model,” Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR 2019), pp.268–275, 2019.
[25] A. Maezawa, “Music boundary detection based on a hybrid deepmodel of novelty, homogeneity, repetition and duration,” Proceedings of the 2018 IEEE International Conference on Acoustics,
Speech and Signal Processing (IEEE ICASSP 2019), pp.206–210,
2019.
[26] G. Sargent, F. Bimbot, and E. Vincent, “Estimating the structuralsegmentation of popular music pieces under regularity constraints,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol.25, no.2, pp.344–358, 2017.
[27] T. Cheng, J.B.L. Smith, and M. Goto, “Music structure boundary detection and labelling by a deconvolution of path-enhanced self-similarity matrix,” Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE ICASSP
2018), pp.106–110, 2018.

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.9 SEPTEMBER 2023

1608

[28] J.B.L. Smith and M. Goto, “Using priors to improve estimates ofmusic structure,” Proceedings of the 17th International Society forMusic Information Retrieval Conference (ISMIR 2016), pp.554–
560, 2016.
[29] T. Grill and J. Schlüter, “Music boundary detection using neural networks on combined features and two-level annotations,” Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR 2015), pp.531–537, 2015.
[30] B. McFee and D. Ellis, “Analyzing song structure with spectral clustering,” Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014), pp.405–410, 2014.
[31] G. Peeters and V. Bisot, “Improving music structure segmentation
using lag-priors,” Proceedings of the 15th International Society forMusic Information Retrieval Conference (ISMIR 2014), pp.337–
342, 2014.
[32] H. Grohganz, M. Clausen, N. Jiang, and M. Müller, “Converting
path structures into block structures using eigenvalue decompositions of self-similarity matrices,” Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR
2013), pp.209–214, 2013.
[33] O. Nieto and T. Jehan, “Convex non-negative matrix factorization
for automatic music structure identification,” Proceedings of the2013 IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE ICASSP 2013), pp.236–240, 2013.
[34] F. Kaiser and G. Peeters, “A simple fusion method of state and sequence segmentation for music structure discovery,” Proceedings ofthe 14th International Society for Music Information Retrieval Conference (ISMIR 2013), pp.257–262, 2013.
[35] J. Serrà, M. Müller, P. Grosche, and J.L. Arcos, “Unsupervised detection of music boundaries by time series structure features,” Proceedings of the 26th AAAI Conference on Artificial Intelligence,
pp.1613–1619, 2012.
[36] M. Müller, P. Grosche, and N. Jiang, “A segment-based fitness measure for capturing repetitive structures of music recordings,” Proceedings of the 12th International Society for Music Information
Retrieval Conference (ISMIR 2011), pp.615–620, 2011.
[37] J. Paulus, M. Müller, and A. Klapuri, “State of the art report: Audiobased music structure analysis,” Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR
2010), pp.625–636, 2010.
[38] J. Paulus and A. Klapuri, “Music structure analysis using a probabilistic fitness measure and a greedy search algorithm,” IEEE Transactions on Audio, Speech, and Language Processing, vol.17, no.6,
pp.1159–1170, 2009.
[39] M. Müller and S. Ewert, “Joint structure analysis with applications
to music annotation and synchronization,” Proceedings of the 9thInternational Conference on Music Information Retrieval (ISMIR
2008), pp.389–394, 2008.
[40] M. Goto, “A chorus section detection method for musical audio signals and its application to a music listening station,” IEEE Transactions on Audio, Speech, and Language Processing, vol.14, no.5,
pp.1783–1794, 2006.
[41] M. Cooper and J. Foote, “Summarizing popular music via structural similarity analysis,” Proceedings of the 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA 2003), pp.127–130, 2003.
[42] J. Foote, “Automatic audio segmentation using a measure of audio
novelty,” Proceedings of the 2000 IEEE International Conference on
Multimedia and Expo (IEEE ICME 2000), pp.452–455, 2000.
[43] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Proceedings of the 26th Annual Conference on Neural Information Processing Systems (NeurIPS 2013), pp.3111–3119, 2013.
[44] O. Melamud, J. Goldberger, and I. Dagan, “context2vec: Learning
generic context embedding with bidirectional LSTM,” Proceedingsof the 20th SIGNLL Conference on Computational Natural Language Learning, pp.51–61, 2016.

[45] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-trainingof deep bidirectional transformers for language understanding,” Proceedings of the 2019 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language
Technologies (NAACL-HLT 2019), pp.4171–4186, 2019.
[46] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC Music
Database: Popular, classical and jazz music databases,” Proceedingsof the 3rd International Conference on Music Information Retrieval(ISMIR 2002), pp.287–288, 2002.
[47] L. Yujian and L. Bo, “A normalized Levenshtein distance metric,”
IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol.29, no.6, pp.1091–1095, 2007.
[48] S. Bird, “NLTK: the natural language toolkit,” Proceedings of the
21st International Conference on Computational Linguistics and44th Annual Meeting of the Association for Computational Linguistics (ACL 2006), pp.69–72, 2006.
[49] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol.9, no.8, pp.1735–1780, 1997.
[50] D.J. Berndt and J. Cliﬀord, “Using dynamic time warping to findpatterns in time series,” Proceedings of Workshop on Knowledge
Discovery in Databases, pp.359–370, 1994.
[51] A. Graves, A. Mohamed, and G. Hinton, “Speech recognition withdeep recurrent neural networks,” Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing(IEEE ICASSP 2013), pp.6645–6649, 2013.
[52] S. Ioﬀe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” Proceedingsof the 32nd International Conference on Machine Learning (ICML2015), pp.448–456, 2015.
[53] C. Raﬀel, B. McFee, E.J. Humphrey, J. Salamon, O. Nieto, D. Liang,
and D.P.W. Ellis, “mir eval: A transparent implementation of common MIR metrics,” Proceedings of the 15th International Society
for Music Information Retrieval Conference (ISMIR 2014), pp.367–
372, 2014.
[54] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” Proceedings of the 7th International Conference on Learning
Representations (ICLR 2019), 2019.
[55] M. Fell, E. Cabrio, F. Gandon, and A. Giboin, “Song lyrics summarization inspired by audio thumbnailing,” Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pp.328–337, 2019.

Kento Watanabe
received the B.E. degreein 2013 and Ph.D. degree in 2018, both fromTohoku University. He is currently a Researcher
at the National Institute of Advanced Industrial
Science and Technology (AIST), Japan. His research interests include lyrics information processing, natural language processing, machinelearning, and human computer iteration.

WATANABE and GOTO: A METHOD TO DETECT CHORUS SECTIONS IN LYRICS TEXT

1609

Masataka Goto
received the Doctor ofEngineering degree from Waseda University in
1998. He is currently a Prime Senior Researcher
at the National Institute of Advanced Industrial Science and Technology (AIST), Japan.
Over the past 30 years he has published morethan 300 papers in refereed journals and international conference proceedings and has received 64 awards, including several best paper awards, best presentation awards, the TenthJapan Academy Medal, and the Tenth JSPSPRIZE. He has served as a committee member of over 120 scientific societies and conferences, including as the General Chair of ISMIR 2009 and2014. As the research director, he began the OngaACCEL project in 2016
and the RecMus project in 2021, which are five-year JST-funded research
projects (ACCEL and CREST) related to music technologies.