Lyric Video Analysis Using Text
Detection and Tracking
Shota Sakaguchi1(B) , Jun Kato2 , Masataka Goto2 , and Seiichi Uchida1
1

Kyushu University, Fukuoka, Japan{shota.sakaguchi,uchida}@human.ait.kyushu-u.ac.jp
2
National Institute of Advanced Industrial Science and Technology (AIST),
Tsukuba, Japan{jun.kato,m.goto}@aist.go.jp

Abstract. We attempt to recognize and track lyric words in lyric videos.
Lyric video is a music video showing the lyric words of a song. The main
characteristic of lyric videos is that the lyric words are shown at frames
synchronously with the music. The diﬃculty of recognizing and tracking
the lyric words is that (1) the words are often decorated and geometrically distorted and (2) the words move arbitrarily and drastically inthe video frame. The purpose of this paper is to analyze the motion of
the lyric words in lyric videos, as the ﬁrst step of automatic lyric video
generation. In order to analyze the motion of lyric words, we ﬁrst apply
a state-of-the-art scene text detector and recognizer to each video frame.
Then, lyric-frame matching is performed to establish the optimal correspondence between lyric words and the frames. After ﬁxing the motion
trajectories of individual lyric words from correspondence, we analyze
the trajectories of the lyric words by k-medoids clustering and dynamic
time warping (DTW).
Keywords: Lyric video · Lyric word tracking · Text motion analysis ·Video design analysis

1

Introduction

The targets of document analysis systems are expanding because of the diversity of recent document modalities. Scanned paper documents only with texts
printed in ordinary fonts were traditional targets of the systems. However, recent
advanced camera-based OCR technologies allow us to analyze arbitrary images
and extract text information from them. In other words, we can now treat arbitrary images with text information as documents.
In fact, we can consider videos as a promising target of document analysis
systems. There have already been many attempts to analyze videos as documents [1]. The most typical attempt is caption detection and recognition in video
frames. Another attempt is the analysis of the video from the in-vehicle camera.
By recognizing the texts in the in-vehicle videos, it is possible to collect store
c Springer Nature Switzerland AG 2020
X. Bai et al. (Eds.): DAS 2020, LNCS 12116, pp. 426–440, 2020.
https://doi.org/10.1007/978-3-030-57058-3_30

Lyric Video Analysis Using Text Detection and Tracking

427(a) “YOU KNOW WHERE”(b) “AND THEN SHE”

(c) “I LOVE YOU”(d) “OOH LOOK WHAT YOU MADE ME
DO”

Fig. 1. Frame image examples from lyric video.

and building names and signboard information around the road automatically.
There was also an attempt (e.g., [2] as a classical trial) to recognize the sport
player’s jersey number for identifying individual players and then analyzing theirperformance.
In this paper, we use lyric videos as a new target of document analysis. Lyric
videos are music videos published at internet video services, such as YouTube,for promoting a song. The main characteristic of lyric videos is that they show
the lyric words of the song (almost) synchronously to the music. Figure 1 shows
several frame examples from lyric videos. Lyric words are often printed in various
decorated fonts and distorted by elaborated visual designs; they, therefore, aresometimes hard to read even for humans. Background images of the frames arephotographic images or illustrations or their mixtures and often too complex
to read the lyrics. This means that lyric word detection and recognition for
lyric videos is a diﬃcult task even for state-of-the-art scene text detectors andrecognizers.
In addition, lyric words in lyric videos move very diﬀerently from words inordinary videos. For example, lyric words often move along with arbitrarilyshaped trajectories while rotating and scaling. This property is very diﬀerent
from video captions since they do not move or just scroll in video frames. It isalso diﬀerent from scene texts in videos from the in-vehicle camera; scene texts
move passively according to camera motion, whereas lyric words move actively
and arbitrarily. In fact, lyric words and their motion are designed carefully by thecreator of the lyric video so that the motion makes the video more impressive.
To give a strong impression, lyric words might move like an explosion.

428

S. Sakaguchi et al.

Fig. 2. Our lyric video analysis task. After detecting and recognizing words in video
frames, the words in the lyric are tracked over frames. Finally, the word motion trajectories are classiﬁed into several clusters to understand the trends of lyric word motions.

The purpose of this paper is to analyze the motion of the lyric words inlyric videos. Speciﬁcally, we try to extract and track words in the lyric words
automatically and then group (i.e., clustering) their motions into several types tounderstand their trends. Nowadays, anyone can post her/his music with video.
This research will contribute to those non-professional creators to design theirown lyric videos. In the future, it will also be possible to realize automatic lyric
video generation, by extending this work to understand the relationship between
lyric word motion and music beat or music style.
For the above purpose, we develop a lyric word detection and tracking
method, where lyric information is fully utilized. As noted before, lyric words
are heavily decorated and shown on various background images. Even stateof-the-art scene text detectors and recognizers will have miss-detections (falsenegatives), false-detections (false-positives), and erroneous word recognition
results. We, therefore, utilize the lyric information as metadata for improving
detection and tracking performance. As shown in Fig. 2, we ﬁrst apply state-ofthe-art scene text detectors and recognizers to each video frame. Then, dynamic
programming-based optimization is applied to determine the optimal matching
between the video frame sequence and the lyric word sequence. We call this novel
technique lyric-frame matching. Then, using the lyric-frame matching result asa reliable correspondence, spatio-temporal search (tracking and interpolation) is
performed for ﬁxing the motion trajectory of each lyric word.
The main contributions of this paper are summarized as follows:– First, to the authors’ best knowledge, this is the ﬁrst trial of detecting andthen tracking lyric words in lyric videos.

Lyric Video Analysis Using Text Detection and Tracking

429– Second, we propose a lyric-frame matching technique where lyric informationof the target music is fully utilized as metadata for accurately determining
the most conﬁdent frame where a lyric word appears. Dynamic programmingand state-of-the-art text detection and recognition techniques are used as themodules of the proposed technique.
– Third, we prepared a dataset comprised of 100 lyric videos and attachedground-truth (bounding boxes for lyric words) manually at 10 frames foreach of 100 videos. This eﬀort enables us to make a quantitative evaluationof our detection and tracking results.
– Fourth, as an analysis of lyric word motions, we grouped them into several representative word motions by k-medoid clustering. From the clusteringresult, we could represent each lyric video by a histogram of representative
motions.

2

Related WorkTo the authors’ best knowledge, this paper is the ﬁrst trial of analyzing lyric
videos as a new document modality. We, however, still can ﬁnd several related
tasks, that is, video caption detection and recognition, and text tracking. In thefollowing, we review the past attempts on those tasks, although the readers also
refer to a comprehensive survey [1, 3]. It should be noted that the performanceon all those tasks has drastically been improved by the recent progress of scene
text detection and recognition technologies. Since the attempts on scene text
detection and recognition are so huge (even if we limit them only to the recentones), we will not make any review on the topic. A survey [3] will give its good
overview.
Caption detection and recognition is a classical but still hot topic of document
image analysis research. Captions are deﬁned as the texts superimposed on video
frames. Captions, therefore, have diﬀerent characteristics from scene texts. We
can ﬁnd many attempts on this task, such as [4–13]. Most of them deal withthe static captions (i.e., captions without motions), while Zedan [10] deals withmoving captions; they assume the vertical or horizontal scrolling of caption text,
in addition to the static captions.
Rather recently, scene text tracking and video text tracking [14–22] have also
been tried, as reviewed in [1]. Each method introduces its own word tracker. For
example, Yang [19] uses a dynamic programming-based tracker to have an optimal spatio-temporal trajectory for each word. A common assumption underlying
those attempts is that text motions in video frames are caused mainly by camera
motion. Therefore, for example, neighboring words will move to similar directions. We also ﬁnd “moving MNIST” [23] for a video prediction task, but it justdeals with synthetic videos capturing two digits are moving around on a uniformbackground.
Our trial is very diﬀerent from those past attempts at the following threepoints at least. First, our target texts move far more dynamically than the texts
that the past attempts assume. In fact, we cannot make any assumption on the

430

S. Sakaguchi et al.

location, appearance, and motion of the words. Second, our trial can utilize thelyric information as the reliable guide of text tracking and therefore we newly
propose a lyric-frame matching technique for our task. Third, our main focus is
to analyze the text motion trajectory from a viewpoint of video design analysis,
which is totally ignored in the past text tracking attempts.

3

Lyric Video Dataset

We collected 100 lyric videos according to the following steps. First, a list of
lyric videos is generated by searching YouTube with the keyword “oﬃcial lyric
video”1 . The keyword “oﬃcial” was added not only for ﬁnding videos with longterm availability and but also for ﬁnding videos created by professional creators.
Videos on the list were then checked visually. A video only with static lyric words(i.e., a video whose lyric words do not move) is removed from the list. Finally,
the top-100 videos on the list are selected as our targets2 .
The collected 100 lyric videos have the following statistics. Each video iscomprised of 5,471 frames (3 min 38 s) on average, 8,629 frames (5 min 44 s) at
maximum, and 2,280 frames (2 min 33 s) at minimum. The frame image size is
1,920 × 1,080 pixels. The frame rate is 12, 24, 25, and 30 fps on 1, 58, 25, and 16
videos, respectively. The lyrics are basically in English. Each song is comprisedof 338 lyric words on average, 690 words at maximum, and 113 at minimum.
Figure 3 shows four examples showing the variations of the number of lyric
words in a frame. Figure 3 (a) shows a typical frame with lyric words. It is quite
often in lyric videos that a phrase (i.e., consecutive lyric words) is shown over
several frames. It is rather rare that only a single lyric word is shown in a oneby-one manner synchronously with the music. We also have frames without any
lyric word, like (b). Those frames are often found in introduction parts, interlude
parts, and ending parts of lyric videos. Figures 3 (c) and (d) contain many words
in a frame. In (c), the same word is repeatedly shown in a frame like a refrain of asong. The words in (d) are just decorations (i.e., they belong to the background)and not relating to the lyrics.
For a quantitative evaluation in our experiment, we attach the boundingboxes for lyric words manually at 10 frames for each lyric video. The 10 frames
were selected as follows; for each video, we picked up frames every three seconds
and then select 10 frames with the most word candidates detected automaticallyby the procedure of Sect. 4.1. The three-second interval is necessary to avoid theselection of the consecutive frames. If a lyric word appears multiple times in aframe (like Fig. 3 (c)), the bounding box is attached to each of them. For therotated lyric words, we attached a non-horizontal bounding box3 . Consequently,
we have 10 × 100 = 1, 000 ground-truthed frames with 7,770 word boundingboxes for the dataset.
1

The search was performed on 18th July 2019.
For the URL list of all the videos and their annotation data, please refer to https://github.com/uchidalab/Lyric-Video.
3
To attach non-horizontal bounding boxes, we used the labeling tool roLabelImg athttps://github.com/cgvict/roLabelImg.
2

Lyric Video Analysis Using Text Detection and Tracking

431(a) Showing lyric words.

(b) No lyric word (interlude).

(c) Duplicated lyric words (like refrains).

(d) Words unrelated to the lyrics.

Fig. 3. Variations of the spatial distribution of lyric words in a video frame.

4

Lyric Word Detection and Tracking by Using Lyric
Information

In this section, we will introduce the methodology to detect and track lyric words
in a lyric video. The technical highlight of the methodology is to fully utilize thelyric information (i.e., the lyric word sequence of the song) for accurate tracking
results. Note that it is assumed that the lyric word sequence is provided for eachlyric video as metadata.
4.1

Lyric Word Candidate Detection

As the ﬁrst step, lyric word candidates are detected by scene text detectors, as
shown in the left side of Fig. 4 (a). Speciﬁcally, we use two pretrained state-of-theart detectors, PSENet [24] and CRAFT [25]. Then, each detected bounding box is
fed to a state-of-the-art scene text recognizer; we used TPS-Resnet-BiLSTM-Attn,
which achieved the best performance in [26]. If two bounding boxes from those two
detectors have 50% overlap and the same recognition result, they are treated as the
duplicated bounding boxes and thus one box is removed from the later process.
4.2

Lyric-Frame Matching

After the detection, lyric-frame matching is performed to ﬁnd the correspondence
between the given lyric word sequence and the frame sequence. The red pathon the right side of Fig. 4 (a) shows the matching path showing the optimalcorrespondence. Assume the video and its lyrics are comprised of T frames andK words, respectively, and t ∈ [1, T ] and k ∈ [1, K] are their indices, respectively.

432

S. Sakaguchi et al.

If the path goes through the node (k, t), the frame t is determined as the most
conﬁdent frame for the kth lyric word. Note that the other frames where thesame kth lyric word appears will be determined later by the tracking process ofSect. 4.3.
The path is determined by evaluating the distance D(k, t) between the kth
word and the frame t. The circled number in Fig. 4 (a) represents D(k, t). A

Fig. 4. The detail of lyric word detection and tracking. (Color ﬁgure online)Lyric Video Analysis Using Text Detection and Tracking

433

smaller value of D(k, t) means that the kth lyric word will be found at t with ahigh probability. The distance is calculated as D(k, t) = minb∈Bt d(k, b), where
Bt is the set of bounding boxes (i.e., words) detected at t and d(k, b) is the edit
distance function between the kth lyric word and the bth detected word at t. If
D(k, t) = 0, the kth lyric word is correctly detected at t without misrecognition.
With the distance {D(k, t)|∀k, ∀t}, the globally optimal lyric-frame matching is obtained eﬃciently by dynamic programming (DP), as the red path ofFig. 4 (a). In the algorithm (so-called the DTW algorithm), the following DP
recursion is calculated at each (k, t) from (k, t) = (1, 1) to (K, T );g(k, t) = D(k, t) +min

t−Δ≤t <tg(k − 1, t ),where g(k, t) becomes the minimum accumulated distance from (1, 1) to (k, t).
The parameter Δ speciﬁes the maximum skipped frames on the path. In thelater experiment, we set Δ = 1, 000 and it means that we allow about 40-s skip
for videos with 24 fps. The computational complexity of the DTW algorithm isO(ΔT K).
It should be emphasized that this lyric-frame matching process with the
lyric information is mandatory for lyric videos. For example, the word “the” willappear in the lyric text many times; this means that there is a large ambiguityof the spatio-temporal location of a certain “the”. We, therefore, need to fully
utilize the sequential nature of not only video frames but also lyric words by the
lyric-frame matching process for determining the most conﬁdent frame for eachlyric word.
4.3

Tracking of Individual Lyric Words

Although the kth lyric word is matched only to a single frame t by the above
lyric-frame matching step, the word will also appear in the neighboring frames oft. We, therefore, search those frames around the tth frame, as shown in Fig. 4 (b).
This search is simply done by evaluating not only spatio-temporal similarity but
also word similarity to the kth word, at the neighboring frames of t. If bothsimilarities are larger than thresholds at t , we assume the same kth word is also
found at t .
Finally, an interpolation process, shown in Fig. 4 (c), is performed for completing the spatio-temporal tracking process of each lyric word. By the above
simple searching process, the lyric word will be missed at a certain frame whena severe misrecognition and/or occlusion occurs on the word at the frame. In
the example of Fig. 4, the third lyric word “THOUGHT” is determined at theﬁfth frame by the lyric-frame matching process and then found at the third andsixth frames by the search. However, it was not found at the fourth frame due
to a severe misrecognition of the word. If such a missed frame is found, the polynomial interpolation process determines the location of the lyric word at theframe. Figure 4 (d) shows the ﬁnal result of the tracking process for two lyric
words “you” and “ever.”434

S. Sakaguchi et al.

(a) Motion by translation (“DON’T”).

(b) Motion by scaling.

(c) Motion by rotating.

Fig. 5. Successful results of lyric word detection and tracking under variable motion
types.

5

Experimental Results

By applying the proposed method to all frames of the collected 100 lyric videos(about 547,100 frames in total), we had the tracking results for all lyric words(about 33,800 words in total).
5.1

Qualitative Evaluation

Figure 5 shows several successful results of lyric word detection and tracking.
In (a), the word “DON’T”, which moves horizontally, is tracked successfully. The
words “LET” and “HIM” in (a) and all the words in (b) are tracked correctly,
although their size varies. The words in (c) are also tracked successfully evenunder frame-by-frame rotation.

Lyric Video Analysis Using Text Detection and Tracking

435

Fig. 6. The eﬀect of lyric information. Top: The initial word detection results shownas green boxes. Bottom: The ﬁnal tracking result shown as blue boxes. The lyric words
in those frames are “well be alright this time”. (Color ﬁgure online)(a) Heavy distortion by partial occlusion (“CAST”).

(b) Tracking error due to multiple appearances of the same word (“BROKEN”).

Fig. 7. Failure results of lyric word detection and tracking.

Figure 6 conﬁrms the eﬀect of using lyric information at the lyric-frame
matching process and the following tracking process for accuracy improvement.
Since these frames have a complex background (with character-like patterns),
unnecessary bounding boxes are found by the initial word detection step. In
contrast, only the correct lyric words remain after lyric-frame matching andtracking.
Figure 7 shows typical failure cases. The failure of (a) was caused by a heavy
distortion on a word by an elaborated visual design of the video. Speciﬁcally, theword “CAST” is always partially occluded and never detected and recognizedcorrectly over frames. Lyric videos frequently show words which are diﬃcult
to read even by the state-of-the-art word detectors and recognizers (and also

436

S. Sakaguchi et al.

Table 1. Performance of the lyric word detection and tracking. MA: Lyric-frame
matching. TR: Tracking. IN: Interpolation. TP: #True-positive. FP: #False-positive.
FN: #False-negative. P: Precision (%). R: Recall (%). F: F-measure.
MA TR IN TP

FP FN



12

72











TP
TP
2PR
P = TP+FP
R= TP+FN
F= P+R

7,698 85.71

0.93

0.0183

5,513 462 2,257 92.27

70.95

0.80225,547 550 2,223 90.98

71.39

0.8000by a human). The failure of (b) is caused by a refrain of the same word in the
lyrics. Especially in musical lyrics, it is very frequent that the same word appears
repeatedly within a short period. These multiple appearances easily distract our
tracking process.
5.2

Quantitative Evaluation

Table 1 shows the quantitative evaluation result of the performance of the lyric
word detection and tracking, by using 1,000 ground-truthed frames. If a bounding box of a lyric word by the proposed method and the ground-truth boundingbox of the same lyric word have IoU > 0.5, the detected box is counted as a successful result. If multiple detected boxes have an overlap with the same groundtruth box with IoU > 0.5, the detected box with the highest IoU is counted as atrue-positive and the remaining are false-positives.
The evaluation result shows that the precision is 90.98% and therefore thefalse positives are suppressed successfully by the proposed method with the lyricframe matching step and the later tracking step. The interpolation step could
increase TPs as expected, but, unfortunately, also increases FPs and slightly
degrades the precision value.
The recall is about 71% at the best setup4 . There are several reasons that
degrade recall. First, as shown in Fig. 7 (a), the elaborated visual design often
disturbs the word detection and recognition process and degrades the recall. In
other words, we need to develop new text detectors and recognizers that arerobust to the visual disturbances to the words in lyric videos as future work.
The second reason for the low recall is the errors at the lyric-frame matching
step. Since our lyric-frame matching allows the skip of Δ = 1, 000 frames (about40 s) at maximum, to deal with frames without lyrics (interlude). This sometimes
causes a wrong skip of the frames where lyric words actually appear and induce
errors in the matching step. Multiple appearances of the same word (such as“I”) also induce matching errors.
The third reason is the deviation from the oﬃcial lyrics; even in the oﬃcial
lyric videos, the lyrics shown in video frames are sometimes diﬀerent from theoﬃcial lyrics due to, for example, the improvisation of the singer. Since we believe
4

As explained before, only a single frame is determined for each lyric word by the
lyric-frame matching. This results in very low recall (0.93%).

Lyric Video Analysis Using Text Detection and Tracking
#1

#12

#43

#27

(a)
#58

#37

#12

(b)

#27

#1

437

#37 #43#51
distribution by tSNE
#58

(c)

#1

#43

(d)

#12
#58
#37 #43

21 30

(c)
(a)

#27

11 20

(b)

(d)

31 40

41 50

51 70 71 100

#1

#12

11 20

21 30

#27

31 40

#37

41 50

51 70 71 100

Fig. 8. The mean word motion histograms of four video clusters. Each of 60 bins
corresponds to a representative word motion. “11∼20” means the 10 representative
motions for the trajectories with 11∼20 frame length. At the right side, the distributionof the 100 histograms of the individual videos is visualized by tSNE.

the oﬃcial lyrics for lyric-frame matching, such deviations disturb the matchingprocess and cause the degradation of recall.
5.3

Text Motion Analysis by Clustering

Finally, we analyze the motion trajectory of lyric words. As the result of the
proposed lyric word detection and tracking process, we already have the motion
trajectory of individual lyric words. Trend analysis of those trajectories will givea hint about how to design lyric videos. Note that we consider that the trajectories subjected to this analysis are reliable enough because the quantitativeevaluation in Sect. 5.2 shows a high precision.
As the motion trajectory analysis, we ﬁrst determine representative motion
trajectories via simple clustering. Before the clustering, we divide them into six
subsets by their length (11∼20 frames (4336 trajectories), 21∼30 (4293), 31∼40(3797), 41∼50 (3159), 51∼70 (4057), 71∼100 (2660)). Note that 6123 extremely
short (≤10) and 2469 extremely long (>100) trajectories are excluded from theanalysis. Then for each subset, we performed k-medoid clustering with DTW
distances between trajectories. We set k = 10 and thus we have 60 representative
motions.
Using those 60 representative motions, we create a histogram with 60 bins
for each lyric video. Speciﬁcally, like a bag-of-words, all word trajectories in thevideo are classiﬁed into one of 60 representative motions and then we have ahistogram whose qth bin represents the number of the lyric words whose motionis the most similar to the qth representative motion. All histograms are thennormalized to have the same number of votes.
After this histogram-based representation of all videos, we perform anotherclustering (k-means) in order to classify 100 videos into clusters with similar

438

S. Sakaguchi et al.

word motions. According to the Calinski-Harabasz criterion, we have 4 clusters.
Figure 8 shows the mean histogram of those clusters. This means that our 100
lyric videos can be classiﬁed into four types and each of them contains the lyric
words whose motion trend is described by the corresponding histogram. Among
100 videos, 55 belong to (a), 11 to (b), 25 to (c), and 9 to (d).
In all the histograms, prominent peaks are found at #1, #12, #27, #37, and#58 and all of those ﬁve representative motions are static motion (with diﬀerent
frame lengths). This indicates that most lyric words are static. With a closer
inspection, we can ﬁnd the ratios of the ﬁve representative motions are diﬀerent
in each histogram. This indicates that the period of showing a word (without
motion) is diﬀerent by the tempo of the music. For example, (c) has many words
displayed only for a very short (11∼20) period.
Another prominent peak is found at #43 and this represents a horizontal
motion. It is interesting to note that a peak at horizontal motions is only foundat 51∼70; this means that speed of the frequent horizontal motions is almost
constant regardless of lyric videos (and also regardless of their tempo). Anotherobservation is that a video that belongs to (a) will contain wide motion varieties.
For example, the #51 representative motion is a circular motion.

6

Conclusion and Future WorkTo the authors’ best knowledge, this paper is the ﬁrst trial of analyzing lyric
videos as novel and dynamic documents. For this diﬃcult analysis task, wedeveloped a novel technique, called lyric-frame matching, where the temporal
location, i.e., the frame, of each word in the lyrics is determined automaticallyby dynamic programming-based optimization. We experimentally showed thatthe combination of the lyric-frame matching technique and several state-of-theart word detectors and recognizers could detect lyric words with more than 90%
precision and 70% recall on our original 100 oﬃcial lyric video dataset. Although
the recall is lower than the precision, the current tracking performance is alreadyreliable for analyzing the motion trajectories of lyric words for understanding
the text motion design in lyric videos. In fact, we could determine four typicalmotion patterns of lyric videos for our 100 videos.
Since this is the ﬁrst trial of lyric video analysis, we have multiple future
works. First, word detection and recognition performance should be improved
for lyric videos. Since the distortions (including elaborated visual designs) on
lyric texts are often very diﬀerent from those of scene texts. This means it isstill insuﬃcient to apply state-of-the-art scene text detectors and recognizers
to this task. The second and more important future work is to analyze word
motion trajectories in lyric videos. Since the moving words in lyric videos area new target of document analysis, we need to develop many analysis schemes.
Word motion analysis for each music genre is straightforward. A more important analysis scheme is the correlation analysis between word motion and musicsignals, i.e., sounds. We often observe that word motions are strongly aﬀectedby the beat and mood of the music. Our ﬁnal goal is to generate lyric videos

Lyric Video Analysis Using Text Detection and Tracking

439

automatically or semi-automatically from lyric information and music signals,
based on those technologies and analyses. For example, our motion trajectory
analysis could contribute to enabling data-driven motion styles on a tool for
creating lyric videos [27].
Acknowledgment. This work was supported by JSPS KAKENHI Grant Number
JP17H06100.

Videos Shown in the Figures
In the ﬁgures of this paper, the following video frames are shown. For URL, thecommon preﬁx part “https://www.youtube.com/watch?v=” is omitted in thelist. Note that the URL list of all 100 videos and their annotation data can befound at https://github.com/uchidalab/Lyric-Video.
– Fig. 1: (a) MK, 17, NoBAfjvhj7o; (b) Demi Lovato, Really Don’t Care,
EOEeN9NmyU8; (c) Alok, Felix Jaehn & The Vamps, All The Lies,
oc218bqEbAA; (d) Taylor Swift, Look What You Made Me Do, 3K0RzZGpyds;– Fig. 2: Robin Schulz, Sugar, jPW5A JyXCY.
– Fig. 3: (a) Marshmello x Kane Brown, One Thing Right, O6RyKbcpBfw; (b)Harris J, I Promise, PxN6X6FevFw; (c) Rita Ora, Your Song, i95Nlb7kiPo;(d) Green Day, Too Dumb to Die, qh7QJ jLam0;– Fig. 4: Freya Ridings, Castles, pL32uHAiHgU.
– Fig. 5: (a) Dua Lipa, New Rules, AyWsHs5QdiY; (b) Anne-Marie, Then,
x9OJpU7O cU; (c) Imagine Dragons, Bad Liar, uEDhGX-UTeI;– Fig. 6: Ed Sheeran, Perfect, iKzRIweSBLA;– Fig. 7: (a) Imagine Dragons, Natural, V5M2WZiAy6k; (b) Kelly Clarkson, Broken & Beautiful, 6l8gyacUq4w;

References
1. Yin, X.C., Zuo, Z.Y., Tian, S., Liu, C.L.: Text detection, tracking and recognitionin video: a comprehensive survey. In: IEEE TIP (2016)
2. Bertini, M., Del Bimbo, A., Nunziati, W.: Automatic detection of player’s identity
in soccer videos using faces and text cues. In: ACM Multimedia (2006)
3. Liu, X., Meng, G., Pan, C.: Scene text detection and recognition with advances indeep learning: a survey. Int. J. Doc. Anal. Recogn. (IJDAR) 22(2), 143–162 (2019).
https://doi.org/10.1007/s10032-019-00320-5
4. Zhao, X., Lin, K.H., Fu, Y., et al.: Text from corners: a novel approach to detect
text and caption in videos. In: IEEE TIP (2010)
5. Zhong, Y., Zhang, H., Jain, A.K.: Automatic caption localization in compressedvideo. In: IEEE TPAMI (2000)
6. Yang, Z., Shi, P.: Caption detection and text recognition in news video. In: International Congress on Image Signal Processing (2012)
7. Lu, T., Palaiahnakote, S., Tan, C.L., Liu, W.: Video caption detection. Video Text
Detection. ACVPR, pp. 49–80. Springer, London (2014). https://doi.org/10.1007/
978-1-4471-6515-6 3

440

S. Sakaguchi et al.

8. Yang, H., Quehl, B., Sack, H.: A framework for improved video text detection
and recognition. Multimedia Tools Appl. 69(1), 217–245 (2012). https://doi.org/
10.1007/s11042-012-1250-6
9. Zhong, D., Shi, P., Pan, D., Sha, Y.: The recognition of Chinese caption text innews video using convolutional neural network. In: IEEE IMCEC (2016)
10. Zedan, I.A., Elsayed, K.M., Emary, E.: Caption detection, localization and type
recognition in Arabic news video. In: INFOS (2016)
11. Chen, L.H., Su, C.W.: Video caption extraction using spatio-temporal slices. Int.
J. Image Graph. (2018)
12. Xu, Y., et al.: End-to-end subtitle detection and recognition for videos in East
Asian languages via CNN ensemble. Sig. Process. Image Commun. 60, 131–143(2018)
13. Lu, W., Sun, H., Chu, J., Huang, X., Yu, J.: A novel approach for video text
detection and recognition based on a corner response feature map and transferred
deep convolutional neural network. IEEE Access 6, 40198–40211 (2018)
14. Qian, X., Liu, G., Wang, H., Su, R.: Text detection, localization, and tracking incompressed video. Sig. Process. Image Commun. 22, 752–768 (2007)
15. Nguyen, P.X., Wang, K., Belongie, S.: Video text detection and recognition: dataset
and benchmark. In: WACV (2014)
16. Gómez, L., Karatzas, D.: MSER-based real-time text detection and tracking. In:
ICPR (2014)
17. Wu, L., Shivakumara, P., Lu, T., Tan, C.L.: A new technique for multi-oriented
scene text line detection and tracking in video. IEEE Trans. Multimedia 17, 1137–
1152 (2015)
18. Tian, S., Yin, X.C., Su, Y., Hao, H.W.: A uniﬁed framework for tracking based
text detection and recognition from web videos. In: IEEE TPAMI (2017)19. Yang, C., Yin, X.C., Pei, W.Y., et al.: Tracking based multi-orientation scene text
detection: a uniﬁed framework with dynamic programming. In: IEEE TIP (2017)20. Pei, W.Y., et al.: Scene video text tracking with graph matching. IEEE Access 6,19419–19426 (2018)
21. Wang, Y., Wang, L., Su, F.: A robust approach for scene text detection and tracking
in video. In: Hong, R., Cheng, W.-H., Yamasaki, T., Wang, M., Ngo, C.-W. (eds.)
PCM 2018. LNCS, vol. 11166, pp. 303–314. Springer, Cham (2018). https://doi.
org/10.1007/978-3-030-00764-5 28
22. Wang, Y., Wang, L., Su, F., Shi, J.: Video text detection with fully convolutional
network and tracking. In: ICME (2019)23. Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video
representations using LSTMs. In: ICML (2015)
24. Wang, W., Xie, E., Li, X., et al.: Shape robust text detection with progressive scale
expansion network. In: CVPR (2019)25. Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text
detection. In: CVPR (2019)26. Baek, J., Kim, G., Lee, J., et al.: What is wrong with scene text recognition modelcomparisons? dataset and model analysis. In: ICCV (2019)
27. Kato, J., Nakano, T., Goto, M.: TextAlive: integrated design environment for
kinetic typography. In: ACM CHI (2015)