Spotting Parodies: Detecting Alignment Collapse
Between Lyrics and Singing Voice
Tomoki Ariga⋆ , Yosuke Higuchi⋆ , Mitsunori Kanno† ,Rie Shigyo† , Takato Mizuguchi† , Naoki Okamoto† , Tetsuji Ogawa⋆⋆

Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan†DAIICHIKOSHO CO., LTD., Tokyo, Japanariga@pcl.cs.waseda.ac.jp

Abstract—We present a method for detecting parodies inkaraoke singing by evaluating alignment collapse between lyrics
and singing voice. Parody detection is a crucial technique forcatching copyright infringement in online vocal recordings andenhancing the accuracy of singing scoring algorithms. Assuming
ground-truth transcriptions (i.e., lyrics) are fixed for a specificsong, we can use an acoustic model to obtain a forced alignment
between lyrics and the corresponding singing voice. However,if the lyrics are not sung correctly and contain parodies, thealignment may not be estimated accurately due to mismatchesin acoustic features. Our aim is to utilize this alignment collapse
to create a robust and effective system for detecting parodies in
singing voice. To this end, we explore various metrics to assess thedegree of collapse in the estimated alignments. We also design
metrics robust against singers’ variations, such as differencesin rhythm arrangements. In our experiments, we construct aJapanese singing voice dataset for training an acoustic model
specific to the singing domain, which is essential for estimating
proper alignments. The results demonstrate that the proposedsystem, adopting an edit-distance-based metric, is highly effective
in detecting parodies, achieving the area under the curve (AUC)of 0.963.
Index Terms—Parody detection, singing voice, forced alignment, karaoke

I. I NTRODUCTION
The use of parody, which involves modifying the lyricsof copyrighted songs, is not recognized as permissible, andposting such a modified song on the internet may be considered copyright infringement. This is a particularly serious
issue for platform providers like YouTube, as they may beheld liable for users uploading parodies, which could result in
the unauthorized spread of pirated content. One naive way to
avoid this problem is to have each piece of content checked
by a human listener, but it is not always feasible due toincreasing operation costs. Hence, there is a high demand foran automated system that can detect parodies in singing voice.
More practically, parody detection can also be useful inassessing singing performances, such as determining whether
a karaoke singer is following the correct lyrics or not. The
commonly used automatic assessment algorithms, including
perceptual evaluation of singing quality (PESnQ) [1], are often
rely solely on acoustic features such as pitch, rhythm, andvibrato. This means that even a humming performance canachieve a high score. By incorporating parody detection, thequality of the scoring system could be significantly enhanced.

ISBN: 978-9-4645-9360-0

parody partFig. 1. Example of forced alignment results for singing voice that is singing
incorrect lyrics (Pred.). Compared with reference result (Ref.), correctly sungparts (symbols c1 , · · · , c3 and c9 , · · · , c12 ) have acceptable alignments.
Other parts with parodies have collapsed alignments due to mismatches withoriginal lyrics (symbols c4 , · · · , c8 ) and speech.

Assuming ground-truth transcriptions (i.e., lyrics) are always fixed for a specific song, one can train an acoustic
model to create a forced alignment between lyrics and thecorresponding singing voice. Previous studies have demonstrated the effectiveness of analyzing forced alignment resultsfor detecting misreads and mispronunciations [2], [3], as well
as for evaluating pronunciation [4]. These studies have also
highlighted that the alignment can be disrupted when there
is a mismatch between the labels and the speech. Figure 1
illustrates an example of forced alignment results between
an incorrectly sung singing voice and its original lyrics. As
reported in other tasks, alignments for the parody parts appear
to be collapsed, while accurately sung parts have precise alignments. This has motivated us to explore parody detection byassessing the degree of collapse in the estimated alignments.
To quantify the degree of collapse, the estimated alignments can be compared with the ground-truth alignments in
a supervised manner. This allows for the evaluation of thealignment precision between singing voice and lyrics [5]–[10]. As the metric for measuring the alignment differences,prior studies have used the mean absolute error between thepredicted and correct boundaries of words [8]. However, sucha metric may not be robust against variations in singing, as it
only considers the absolute timing differences. For instance,a karaoke singer can add arrangements to their performanceby adjusting their singing timing, by changing the timing toslightly earlier or later and altering the duration to shorter or
longer. As our goal is to identify parodies in singing voices
performed by users with varying characteristics, it is crucial
to explore alternative approaches that can evaluate alignment
collapse while accounting for individual differences in aspects
such as rhythm arrangements and timing shifts.

286

EUSIPCO 2023

c1

T

X

T

T̂

c2
c1

T̂
c1

c2

ts1

c3

te1 ts2

te2

c1

t̂s1

ts3

c2

t̂e1

t̂s2 t̂e2

c2|t̂s1 − ts1|

te3

c3
c3

|t̂s3 − ts3||t̂s2 − ts2|

Fig. 3. Mean absolute error (MAE) based on word boundaries.

c3

t̂s3

t̂e3

T

Fig. 2. Example ground-truth alignment T and estimated alignment T̂between input singing voice sample X and the corresponding lyrics C. Eachalignment includes start and end timings of the n-th symbol cn .

c1

c2
c1

T̂
l1

c3

c2

c3

l2

l3

Fig. 4. Percentage of correctly labelled intervals (Perc).

In this work, we develop a reliable and effective parody detection system for singing voice. The proposed approach involves analyzing forced alignment results between
the singing voice and the corresponding lyrics. We investigate
and compare various metrics to assess the degree of collapse in
the estimated alignments. Accounting for variations in users’singing styles, we also design a robust metric that effectively
absorbs differences in singing timing. Since the quality offorced alignment performance is crucial to our approach, it
is necessary to train an acoustic model specifically for thesinging voice domain [11]–[13]. To achieve this, we newly
construct a monophonic Japanese singing voice database,which includes 198 hours of karaoke singing voices recorded
in a real environment.
The rest of the paper is organized as follows. Section II
presents the proposed parody detection system, introducing
different metrics for assessing the alignment collapse. Section III presents experimental results of the proposed systemevaluated on the real singing voice dataset. Finally, Section IV
concludes with a discussion of the potential limitations of the
proposed system.

3) Evaluate the alignment collapse by comparing the estimated alignment T̂ with the reference alignment T .
4) Determine if the target singing voice includes a parody
by comparing the evaluated score from Step 3 to apredefined threshold.
Note that our parody detection system does not provide theexact location of where a parody is sung, but it determines
whether a short phrase (≈ 10 seconds) of a singing voicecontains a parody or not.
The following subsections describe metrics used in Step 3
to assess the degree of collapse in the estimated alignmentT̂ . We first introduce two existing approaches, based on themean absolute error (MAE) [8] and the percentage of correctly
labelled intervals (Perc) [5]. We then design two other metrics
that take into account variations in singing timing.
A. Mean Absolute Error (MAE)
Figure 3 illustrates the conventional metric based on MAE.
MAE computes the absolute time difference between theestimated and ground-truth boundaries of the n-th symbol cn :II. P ROPOSED PARODY D ETECTION S YSTEMNIn Figure 2, we explain our problem setting for constructing
a parody detection system. Given a singing voice sample X,
let C = (cn ∈ V|n = 1, · · · , N ) be the corresponding
lyrics, consisting of N symbols (i.e., Japanese characters) in a
vocabulary V. We define T = {⟨tsn , ten ⟩|n = 1, · · · , N } as theground-truth alignment, where tsn and ten indicate the start andend time stamps for uttering the n-th symbol cn . Similarly, wedefine T̂ = {⟨t̂sn , t̂en ⟩|n = 1, · · · , N } as the estimated timings
from forced alignment. The primary goal of our system is todetermine whether X contains any parody by comparing thealignments T and T̂ . Taking Figure 2 as an example, it appears
that the estimated alignment for c2 is somewhat collapsed andmay not be sung correctly, similar to the parody part depicted
in Figure 1. We explore an effective metric for detecting suchan alignment error.
The proposed parody detection system is constructed using
the following steps.
1) Train an acoustic model.
2) With the acoustic model, obtain forced alignment resultsT̂ between a target singing voice X and its lyrics C.

Dmae =1 X s|t̂ − tsn |.
N n=1 n(1)

MAE is effective at computing the differences between timings
that are significantly unaligned or exactly aligned (e.g., c2 andc3 in Figure 3). However, if there are persistent delayed or
hastened parts (e.g., c1 in Figure 3), such as those caused by
the recording environment or users’ arrangements, MAE willaccumulate these differences, leading the system to produce
incorrect decisions.
B. Percentage of Correctly Labelled Intervals (Perc)
Figure 4 illustrates the conventional metric based on Perc.
Perc calculates the ratio of accurately aligned intervals to theoverall input length as
ln = min(ten , t̂en ) − max(tsn , t̂sn ),(2)

1
max(ln , 0),|X| n=1(3)

Dperc =

N
X

where ln is the overlap length between the estimated andground-truth singing timings. Perc differs from MAE in that

287

T

c2

c1

ts1

te1

te2

c1

T̂

t̂s1

ts3

te3

T̂

c3

c2

t̂e1

T

c3

ts2

t̂s2 t̂e2

t̂s3

c1

c3

c2

c1

c2

c3

deletet̂e3insertdpe
dde

substituteFig. 5. Combined position and duration errors (PE+DE).

it considers the duration of a singing part between the alignments. Like MAE, Perc is also limited in its ability to handle
consistently delayed or hastened timings.
C. Combined Position and Duration Errors (PE+DE)

Fig. 6. Example of how edit distance is used to detect collapsed alignment.
Here, estimated alignment T̂ has two-frame delay. This can be simply
adjusted by deleting and inserting “<sp>” at onset of c1 and offset of c3 .
Estimated segment for c2 is extremely short, and it requires a large number ofsubstitutions to match the reference segment. These properties can be effective
at identifying alignment collapse while avoiding unintended accumulation oftiming shift errors.

In order to address the issue presented in the MAE and Perc
metrics, we design a metric that takes into account variations
in timing shifts. To this end, we define the position error (PE)and duration error (DE) between the alignments, as depicted in
Figure 5. For the alignments corresponding to the n-th symbol
in T and T̂ , the position and duration errors are calculated ast̂s + t̂en
tsn + ten
− n
,
2
2
s
s
e
e
dde
n = tn − t̂n − (tn − t̂n ) .

dpe
n =

TABLE ID ETAILS OF T EST S ET. E ACH SONG CONTAINS ABOUT 10 UTTERANCESOF PARODY SINGING .

(4)Song Name

Durations / Utt. [s]# Utts. (w/ parody)

Song-A
Song-BSong-CSong-D
Song-E

13.1
12.8
15.0
11.8
13.4

370 (12)326 (13)516 (11)324 (10)84 (9)(5)

The position error dpe measures positional deviations basedon the distance between the center points of each segment
pair. The duration error dde measures the difference in duration
between each segment. The two errors are combined as
(
e
s
dpe
if dpenn > tn − tn ,(6)dn =otherwise.
ddenlyrics. Let Y = (yt ∈ V ∪ {“<sp>”}|t = 1, · · · , |X|) andŶ = (ŷt ∈ V ∪ {“<sp>”}|t = 1, · · · , |X|) denote the framelevel symbols from ground-truth and estimated alignments,
respectively. Here, the t-th symbol is derived as(cn
if tsn ≤ t ≤ ten ,(8)
yt =“<sp>” otherwise,

The first statement in Eq. (6) is expected for a case, such as
c1 in Figure 5, where the position error of a target symbol
is higher than the ground-truth duration. This indicates that
there is no overlap between the ground-truth and estimatedsegments, which can be an obvious cue for detecting alignment
collapse. If the position difference between segments is not
significant, such as c2 depicted in Figure 5, we solely consider
the duration error, as represented by the second statement of
Eq. (6). This indicates that the segments overlap, and only the
duration error is crucial in detecting errors, while any minor
positional differences can be disregarded. The final score is
obtained by calculating dn for all symbols as

where “<sp>” is a space symbol indicating silence. The
alignment difference between T and T̂ is computed based on
the edit distance (i.e., Levenshtein distance) using the framelevel sequences Y and Ŷ asN

Dpe-de =1 X
dn .
|X| n=1(7)

By computing the score using Eq. (7), we expect the metric
to be robust against minor timing shifts.
D. Edit DistanceAs an alternative approach, we also explore using the edit
distance to measure the dissimilarity between the alignments.
By establishing an alignment between a singing voice andits lyrics, it is possible to obtain frame-level symbols thatindicate which frame corresponds to which symbols in the

Ded =

1
EditDistance(Y, Ŷ ).
|X|

(9)

Figure 6 illustrates how the edit distance reflects the differences in the alignments. The edit distance is capable ofaccommodating variations in the starting and ending timingsof singing parts. This is achieved through a minimal set ofoperations, such as inserting the onset or deleting the offset,
which allow for a constant shift in the overall timing of analignment. In other metrics, even constant timing deviations
may accumulate a penalty for each symbol pair, resulting inthe song being unduly considered a parody. If the estimatedalignment contains a collapsed portion, a significant numberof space symbol insertions (or target symbol deletions) may
occur, thereby increasing the potential for detecting parody.
III. E XPERIMENTS
A. Dataset
We conducted experiments using an in-house monophonicsinging voice dataset, which was constructed from karaoke288

singing recordings by real users. All of the singing voices
were recorded in a karaoke box1 using a single dynamic
microphone, resulting in slight reverberation and backgroundmusic.
Training set. The training set consisted of 198 hours
of singing voices, featuring 4461 songs performed by 1318
amateur singers. The majority of the songs were in Japanese,and a few contained English lyrics. To train the acoustic
model, we segmented each singing voice in the training set
into phrases lasting from 1 to 10 seconds.
Test set. Table I provides details of the test set, which
included one-phrase karaoke recordings of five Japanese songs(Song-*) sung by 1503 amateur singers. While most of thephrases in each song were sung correctly, some were sungwith parodies. Each song contained approximately 10 samplesof parody singing, which included three different types ofmodified lyrics.
The ground-truth alignment T was obtained from subtitle
information provided by a karaoke system. Note that this subtitle information was labeled manually and may not preciselyreflect the actual alignment. This issue can be also mitigatedby the metrics designed to be robust against timing shifts.

TABLE IIP ERFORMANCE OF DETECTING PARODY EVALUATED USING AREA UNDER
THE CURVE (AUC) FOR FIVE SONGS IN TEST SET.
Song Name
Metric
MAE
Perc
PE+DE
Edit Distance

A

B

C

D

E

Average

0.928
0.841
0.952
0.942

0.865
0.929
0.894
0.904

0.808
0.726
0.851
0.995

0.960
0.964
0.983
0.994

0.880
0.880
0.873
0.979

0.890
0.868
0.911
0.963

B. Acoustic Model
We constructed a DNN-HMM-based acoustic model using
the Kaldi speech recognition toolkit [14] with the “nnet1”
setup [15]. To process the model input, we first extracted 13dimensional mel-frequency ceptral coefficients (MFCCs) andtheir delta parameters using a 25ms analysis window and a
10 ms window shift, which were reduced to 40-dimensionalfeatures through linear discriminant analysis (LDA). We thentransformed the features into a speaker-normalized space using semi-tied covariance (STC) matrices and feature-space
maximum likelihood linear regression (fMLLR), resulting in
40-dimensional acoustic features. The DNN was pre-trainedusing restricted Boltzmann machines [16], and then trained toclassify triphone states from each feature frame.
Although some recent studies focus on large pre-trainedmodels (e.g., wav2vec2.0 [17]) for singing voice processing [18], [19], we chose the traditional triphone-based DNNHMM system. Alignment estimation is a relatively simpler
task compared to others such as automatic lyric transcription.
In fact, [20] has reported that a GMM-HMM system iscapable of estimating alignments with human-level performance. Moreover, such a light-weight system can be readily
employed in real-time applications, including online singing
voice scoring.
C. Evaluated Items
The purpose of this experiment was twofold: 1) to identify
effective metrics for detecting alignment collapse, and 2) toevaluate the performance of the proposed parody detection
system. To this end, we compared the following alignment
metrics described in Sec. II:
1 A small private room containing karaoke equipment.

Fig. 7. ROC curve for Song-C, comparing four evaluation metrics of parody
detection experiments on 516 correct and 11 parody singing voices. Hightrue positive rate and low false positive rate are required for parody detection
system.

MAE [8]: Conventional metric calculated using Eq. (1);Perc [5]: Conventional metric calculated using Eq. (3);
• PE+DE: Proposed metric calculated using Eq. (7); and• Edit Distance: Proposed metric calculated using Eq. (9).
To evaluate and compare the alignment metrics for theircapability in parody detection, we calculated the area under
the curve (AUC) based on the score D∗ measured using each
metric.
••D. Results
Table II lists the AUC scores of each alignment metricevaluated on the test set. We observed that even when using
the conventional MAE and Perc metrics, the system achieved
reasonable performance with AUC scores higher than 0.8. This
indicates that analyzing the estimated alignment to identify acollapsed part is a practical approach for detecting parody
in singing voices. Compared to the conventional methods,
the proposed metrics based on PE+DE and Edit Distancedemonstrated superior performance (AUC scores higher than0.9) in parody detection. We attribute this improvement to theshift-robust design intended to accommodate the variations
in singing timing. Among all the metrics, Edit Distanceproduced the best performance, with an impressive average
AUC score of 0.963.
Figure 7 depicts an example ROC curve generated fromthe test sample Song-C. The closer curves are to the top-leftcorner, the better the performance in detecting parody. The

289

curves clearly demonstrate the superiority of Edit Distanceover the other metrics, exhibiting a better trade-off between
true positive and false positive rates.
IV. C ONCLUSIONS AND L IMITATIONS
We developed a system for detecting parodies in singing
voice. Different types of metrics were compared to evaluatealignment collapse between singing voices and the corresponding lyrics, and we designed effective metrics to account forvariations in singing timings. Experimental results on 198
hours of singing voice data demonstrated that the proposed
metrics significantly improved the performance of parody
detection compared to the baselines. Notably, the metrics
based on the edit distance achieved the highest AUC scoreof 0.963, indicating its effectiveness in assessing alignment
collapse.
One limitation of the proposed system is that it detects thepresence of a parody at the phrase level, which is approximately 10 seconds. Therefore, if the duration of parody parts
is significantly small (e.g., only one character is modified ina phrase), the detection performance is likely to be reduced.
One potential solution to this issue is further to divide phrases
into shorter units, such as words.
Another possible limitation to consider is that if the modified part has vowels that are similar to those of the original
part, the alignment may not be significantly collapsed. Thisis because forced alignment relies on an acoustic modelthat is trained on phonemes. In such cases, incorporating
additional acoustic and linguistic information, such as theacoustic likelihood, is expected to enhance the performanceof parody detection.

[10] J. Huang, E. Benetos, and S. Ewert, “Improving lyrics alignment
through joint pitch detection,” in Proc. IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 451–
455.
[11] Y. Ohishi, M. Goto, K. Itou, and K. Takeda, “Discrimination between
singing and speaking voices,” in Proc. ISCA Interspeech, 2005, pp.
1141–1144.
[12] A. Mesaros and T. Virtanen, “Adaptation of a speech recognizer for
singing voice,” in Proc. IEEE European Signal Processing Conference(EUSIPCO), 2009, pp. 1779–1783.
[13] Z. Duan, H. Fang, B. Li, K. C. Sim, and Y. Wang, “The NUS sung andspoken lyrics corpus: A quantitative comparison of singing and speech,”in Proc. Asia-Pacific Signal and Information Processing Association
Annual Summit and Conference (APSIPA ASC), 2013, pp. 1–9.
[14] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel,
M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi speech recognition toolkit,” in Proc.
IEEE Workshop on Automatic Speech Recognition and Understanding(ASRU), 2011.
[15] K. Veselỳ, A. Ghoshal, L. Burget, and D. Povey, “Sequencediscriminative training of deep neural networks,” in Proc. ISCA Interspeech, 2013, pp. 2345–2349.
[16] G. E. Hinton, “A practical guide to training restricted boltzmann
machines,” Neural Networks: Tricks of the Trade: Second Edition, pp.
599–619, 2012.
[17] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A
framework for self-supervised learning of speech representations,” Proc.
Advances in Neural Information Processing Systems (NeurIPS), vol. 33,
pp. 12 449–12 460, 2020.
[18] L. Ou, X. Gu, and Y. Wang, “Transfer learning of wav2vec 2.0 forautomatic lyric transcription,” in Proc. International Society for Music
Information Retrieval Conference (ISMIR), 2022.
[19] X. Gao, X. Yue, and H. Li, “Self-transriber: Few-shot lyrics transcription
with self-training,” arXiv preprint arXiv:2211.10152, 2022.
[20] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger,“Montreal forced aligner: Trainable text-speech alignment using Kaldi,”
in Proc. ISCA Interspeech, 2017, pp. 498–502.

R EFERENCES[1] C. Gupta, H. Li, and Y. Wang, “Perceptual evaluation of singing quality,”in Proc. Asia-Pacific Signal and Information Processing Association
Annual Summit and Conference (APSIPA ASC), 2017, pp. 577–586.
[2] A. Lee and J. Glass, “A comparison-based approach to mispronunciation
detection,” in Proc. IEEE Spoken Language Technology Workshop (SLT),2012, pp. 382–387.
[3] ——, “Mispronunciation detection without nonnative training data,” inProc. ISCA Interspeech, 2015, pp. 643–647.
[4] C. Gupta, H. Li, and Y. Wang, “Automatic pronunciation evaluation of
singing,” in Proc. ISCA Interspeech, 2018, pp. 1507–1511.
[5] H. Fujihara, M. Goto, J. Ogata, and H. G. Okuno, “LyricSynchronizer:
Automatic synchronization system between musical audio signals andlyrics,” IEEE Journal of Selected Topics in Signal Processing, vol. 5,
no. 6, pp. 1252–1261, 2011.
[6] M. Mauch, H. Fujihara, and M. Goto, “Integrating additional chord information into HMM-based lyrics-to-audio alignment,” IEEE Transactions
on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 200–
210, 2011.
[7] Y.-R. Chien, H.-M. Wang, and S.-K. Jeng, “Alignment of lyrics withaccompanied singing audio based on acoustic-phonetic vowel likelihood
modeling,” IEEE/ACM Transactions on Audio, Speech, and Language
Processing, vol. 24, no. 11, pp. 1998–2008, 2016.
[8] D. Stoller, S. Durand, and S. Ewert, “End-to-end lyrics alignment for
polyphonic music using an audio-to-character recognition model,” inProc. IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2019, pp. 181–185.
[9] C. Gupta, E. Yılmaz, and H. Li, “Automatic lyrics alignment andtranscription in polyphonic music: Does background music help?” in
Proc. IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2020, pp. 496–500.

290