A COMPUTATIONAL EVALUATION FRAMEWORKFOR SINGABLE LYRIC TRANSLATION
Haven Kim 1
Kento Watanabe 2
Masataka Goto 2
Juhan Nam 1
1
Graduate School of Culture Technology, KAIST, South Korea
2
National Institute of Advanced Industrial Science and Technology (AIST), Japankhaven@kaist.ac.kr, kento.watanabe@aist.go.jp, m.goto@aist.go.jp, juhan.nam@kaist.ac.kr

ABSTRACT
Lyric translation plays a pivotal role in amplifying the
global resonance of music, bridging cultural divides, andfostering universal connections. Translating lyrics, unlike
conventional translation tasks, requires a delicate balance
between singability and semantics. In this paper, we present
a computational framework for the quantitative evaluationof singable lyric translation, which seamlessly integrates
musical, linguistic, and cultural dimensions of lyrics. Our
comprehensive framework consists of four metrics that measure syllable count distance, phoneme repetition similarity,
musical structure distance, and semantic similarity. To
substantiate the efficacy of our framework, we collecteda singable lyrics dataset, which precisely aligns English,
Japanese, and Korean lyrics on a line-by-line and sectionby-section basis, and conducted a comparative analysis
between singable and non-singable lyrics. Our multidisciplinary approach provides insights into the key componentsthat underlie the art of lyric translation and establishes asolid groundwork for the future of computational lyric translation assessment.
1. INTRODUCTION
Translating lyrics is a prevailing method of enhancing the
global appeal and allure of music across a multitude ofgenres, such as theatre music, animation music, pop music,
etc [1]. Furthermore, recent advancements in media technology have facilitated the exchange of intercultural products
and globalized fandom culture, resulting in an increase in
the popularity of user-translated lyrics across diverse social
media platforms [2].
Despite its popularity, lyric translation is acknowledged
as a challenging field, requiring an interdisciplinary approach [2]. As early as 1915, it was suggested that an ideal
lyric translator should possess expertise in both linguistics and music, highlighting the need for a comprehensive
understanding of the principles and techniques used in translation studies, coupled with a background in musicology [3].
© H. Kim, K. Watanabe, M. Goto, and J. Nam. Licensed
under a Creative Commons Attribution 4.0 International License (CC BY
4.0). Attribution: H. Kim, K. Watanabe, M. Goto, and J. Nam, “A
Computational Evaluation Framework for Singable Lyric Translation”,
in Proc. of the 24th Int. Society for Music Information Retrieval Conf.,
Milan, Italy, 2023.

774

Moreover, it is crucial to understand the cultural context
of each language, such as different strategies employed forforming rhymes [4]. Because of these challenges, the systematic analysis and evaluation of lyric translation remain
an under-researched topic of study. Thus far, only a few
have proposed guidelines for scoring the quality of translated lyrics [4, 5]. While these principled approaches have
proven successful, they lack automation. Consequently,
despite the growing interest in the development of neurallyric translation, its evaluation has predominantly reliedon human evaluation, making the evaluation process timeconsuming, unreliable, and subjective [6–8].
Our study aims to computationally analyze and evaluate
lyric translation based on a comprehensive understanding oflyrics that accounts for their musical, linguistic, and culturalelements. Unlike prior research that only proposed rhymescoring guidelines applicable to English [4], our framework
is extendable to Japanese and Korean. Though our framework may be limited in its application to specific languages,
we strive to provide valuable insights into establishing distinct evaluation rules for phoneme repetition in diverse
languages. Our comprehensive framework employs a multifaceted evaluation approach that examines lyric translationfrom four distinct perspectives: syllable counts, phoneme
repetition, musical structure, and semantics. In the remainder of this paper, we explicate the rationale behind ourselection of these perspectives by delving into the uniquecharacteristics of lyric translation that differentiate it fromgeneral language translation tasks. In addition, we introduce the singable lyrics dataset we collected, which featuresline-by-line and section-by-section alignment of English,
Japanese, and Korean lyrics. Moving forward, we propose
robust evaluation metrics for lyric translation and analyze
the results of our experiments based on the perspectives
mentioned above. Finally, we conclude our paper by reflecting on the profound insights gleaned from our experimentand highlighting possible directions for future research.
2. BACKGROUND
Previous research indicates that linguistic analysis methods designed for standard text may not achieve desired
outcomes when used to examine lyrics [9]. Although automated evaluation metrics, such as n-gram-based [10–12] orneural approaches [13], have proven valuable and effectivein assessing conventional machine translation tasks, theyProceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023

fall short in evaluating lyric translation. This is due to theunique characteristics of lyrics that render the translation
process subject to many constraints and less direct [14].
One of the most significant constraints is the syllable
count. This is because the original and translated lyrics mustmatch the same melody lines, while it is a common practice
to tweak the melody to accommodate minor changes in syllable count [4, 15]. In fact, conveying the same message in
different languages requires vastly different syllable counts.
For example, “Happy New Year” in English consists of
4 syllables, whereas 15 and 9 are required for Japaneseᅢ
(あけましておめでとうございます) and Korean (ᄉ
ᄒ
ᅢᄇ
ᆨᄆ
ᅩ
ᆭᄋ
ᅡ
ᅵᄇ
ᆮᄋ
ᅡ
ᅳᅦ
ᄉᄋ
ᅭ), respectively. For the numericalcomparison, we examined PAWS-X, a dataset that contains
23,659 English sentences paired with human-translated sentences in various languages [16]. The average number ofsyllables per sentence in the dataset is 50.89 for Japanese,
whereas 36.18 per English and 40.40 per Korean. With
these statistics, it can be deduced that Japanese necessitates approximately 41% more syllables than English and
26% more syllables than Korean to express an equivalentmessage. This limitation forces translators to often modify the meaning of original lyrics by adding, omitting, oreven tweaking the message. However, translated lyrics still
aim to capture the theme, mood, and spirit of the originallyrics [17]. Therefore, while original and translated lyrics
need not be semantically identical, they still need to besemantically relevant [4, 18].
It is also crucial to preserve the frequency of phoneme
repetition (e.g., rhyme) in translated lyrics, particularly
when the music demands it [17]. For instance, some
sections, such as choruses, require a substantial degreeof phoneme repetition, while others do not. Moreover,due to the inherent connection between lyrics and music,
lyrics must be arranged in a way that complements the music [19]. As a result, musically similar sections should maintain resembling linguistic features, including the choice ofphonemes and the frequency of phoneme repetition [20].
3. DATASET
Although some websites provide user-translated multilingual lyrics, we found that most of them lack singability, as these translations were focused on delivering themeaning of the original lyrics rather than making them
performable. While there are a few singable translations
available, they are often not aligned on a line-by-line nor
section-by-section basis due to the subjective nature of the
lyric structure that there is no universal agreement on what
to call a line and what to call a section. The absence ofalignment makes it difficult to compare the original lyrics
with their translated versions. To address this issue, wecollected a set of singable lyrics, sourced from either official lyrics of commercial songs or user-translated ones
found on YouTube, meticulously aligned on a line-by-line
basis in English, Japanese, and Korean. This approach ensures that lyrics on the same line share the same melodies.
Moreover, the dataset divides the lyrics into sections, allowing for section-by-section analysis. Alongside the lyrics, itSection#Line#

English (EN)Japanese (JP)Korean (KR)

1

1
2
3
4
5
6

Twinkle, twinkle, little star
How I wonder what you are
Up above the world so highLike a diamond in the sky
Twinkle, twinkle, little star
How I wonder what you are

きらきらひかるおそらのほしよ
まばたきしては
みんなをみてるきらきらひかるおそらのほしよ

ᄇᄍ
ᆫ
ᅡ
ᆨᄇ
ᅡ
ᆫᄍ
ᅡ
ᆨᄌ
ᅡ
ᆨᄋ
ᅡ
ᆫᄇ
ᅳ
ᆯ
ᅧ
ᅡᄅ
ᄋ
ᆷᄃ
ᅳ
ᆸᄀ
ᅡ
ᅦᄇ
ᅵᄎ
ᅵᄂ
ᅦ
ᅥᄍ
ᄉ
ᆨᄒ
ᅩ
ᅡᄂ
ᆯᄋ
ᅳ
ᅦᄉ
ᅥᄃ
ᅩ
ᆼᄍ
ᅩ
ᄃ
ᆨᄒ
ᅩ
ᅡᄂ
ᆯᄋ
ᅳ
ᅦᄉ
ᅥᄃ
ᅩ
ᆫᄍ
ᅡ
ᄇ
ᆨᄇ
ᅡ
ᆫᄍ
ᅡ
ᆨᄌ
ᅡ
ᆨᄋ
ᅡ
ᆫᄇ
ᅳ
ᆯ
ᅧ
ᅡᄅ
ᄋ
ᆷᄃ
ᅳ
ᆸᄀ
ᅡ
ᅦᄇ
ᅵᄎ
ᅵᄂ
ᅦ

2

7
8
9
10
11
12

Twinkle, twinkle, little star
How I wonder what you are
When the blazing sun is gone
When he nothing shines upon
Then you show your little light
Twinkle, twinkle, all the night

きらきらひかるおそらのほしよみんなのうたがとどくといいな
きらきらひかるおそらのほしよ

ᄇᄍ
ᆫ
ᅡ
ᆨᄇ
ᅡ
ᆫᄍ
ᅡ
ᆨᄌ
ᅡ
ᆨᄋ
ᅡ
ᆫᄇ
ᅳ
ᆯ
ᅧ
ᅡᄅ
ᄋ
ᆷᄃ
ᅳ
ᆸᄀ
ᅡ
ᅦᄇ
ᅵᄎ
ᅵᄂ
ᅦ
ᅥᄍ
ᄉ
ᆨᄒ
ᅩ
ᅡᄂ
ᆯᄋ
ᅳ
ᅦᄉ
ᅥᄃ
ᅩ
ᆼᄍ
ᅩ
ᄃ
ᆨᄒ
ᅩ
ᅡᄂ
ᆯᄋ
ᅳ
ᅦᄉ
ᅥᄃ
ᅩ
ᆫᄍ
ᅡ
ᄇ
ᆨᄇ
ᅡ
ᆫᄍ
ᅡ
ᆨᄌ
ᅡ
ᆨᄋ
ᅡ
ᆫᄇ
ᅳ
ᆯ
ᅧ
ᅡᄅ
ᄋ
ᆷᄃ
ᅳ
ᆸᄀ
ᅡ
ᅦᄇ
ᅵᄎ
ᅵᄂ
ᅦ

Table 1. Sample data illustrating the original English lyrics
of “Twinkle, Twinkle, Little Star” and their corresponding
singable translations in Japanese and Korean, aligned on aline-by-line and section-by-section basis.
provides essential metadata such as genre, artist, original
language, and the official status of lyrics. The dataset consists of 162 songs, each having lyrics in the three languages.
It covers a diverse range of genres, including 109 K-pop, 23
animation music (e.g., Disney), 13 J-pop, 10 theatre music,
and more. Table 1 shows sample data.

4. EVALUATING SINGABILITY
Our primary goal is to develop an evaluation frameworkthat automatically assesses the quality of translated lyrics.
One of the most important factors determining the quality is
singability, defined as not only the ability of being sung, butalso the suitability (easiness) of being sung [18]. To ensure
such singability, we aim to provide metrics from three distinct perspectives by making sure that they i) maintain thesong’s melodic integrity, ii) preserve the degree of phoneme
repetition, and iii) consider the underlying musical structure.
To substantiate the reliability of our evaluation metrics,
we conducted a comparative analysis of singable lyrics versus non-singable lyrics based on each proposed evaluation
metric. In all our comparative analyses, we utilized ourdataset for singable lyrics, where official lyrics served asboth source and target lyrics, and unofficial functioned as
only target lyrics. For non-singable lyrics, we obtained
pairs of original singable (source) and human-translated
non-singable (target) lyrics, aligned line-wise and sectionwise, for 3,642 songs from https://lyricstranslate.com/.

SourceTargetSingableNon-singable

EnglishJapanese
Korean0.17 (80 songs)
0.11 (80 songs)

0.74 (1401 songs)
0.48 (620 songs)

Japanese

EnglishKorean0.16 (162 songs)
0.11 (162 songs)

0.39 (589 songs)
0.31 (73 songs)

KoreanEnglishJapanese

0.09 (161 songs)
0.12 (161 songs)

0.20 (702 songs)
0.52 (257 songs)

Table 2. The average line syllable count distance (Dissyl )between source and target languages for singable and nonsingable lyrics.
775

Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023
SectionEnglishJapanese (English translation)

Korean (English translation)phoE(A1 ),
J(A1 ),
K(A1 )

Do you wanna build a snowman?
Come on, let’s go and play!
I never see you anymore
Come out the door
It’s like you’ve gone away

雪だるま作ろう (Let’s build a snowman)
ドアを開けて (Please open the door)
一緒に遊ぼう (Let’s play together)
どうして (Why)
出てこないの? (don’t you come out?)

ᄀᄋ
ᇀ
ᅡ
ᅵᄂ
ᆫᄉ
ᅮ
ᅡᄅ
ᆷᄆ
ᅡ
ᆫᄃ
ᅡ
ᅳ
ᆯᄅ
ᅢ? (Do you wanna build a snowman?)
ᅦᄇ
ᄌ
ᆯᄌ
ᅡ
ᆷᄂ
ᅩ
ᅡᄋ
ᅪᄇ
ᅪ (Please come out)
ᆫᄂ
ᅥ
ᄋ
ᅵᄅ
ᆯᄆ
ᅳ
ᆫᅡ
ᅡ
ᆯ
ᄂᄉ
ᅮᄋ
ᆹᄋ
ᅥ
ᅥ (I can’t meet you)
ᇀᄋ
ᅡ
ᄀ
ᅵᄂ
ᆯᄌ
ᅩ
ᅡ (Let’s play together)
ᅡᄒ
ᄂ
ᆫᄌ
ᅩ
ᅡᄉ
ᆷᄉ
ᅵ
ᆷᄒ
ᅵ
ᅢ (I’m lonely alone)

0.85,
0.73,
0.77

E(B1 ),
J(B1 ),
K(B1 )We used to be best buddies
And now we’re not
I wish you would tell me why!

前は仲良く(We were close before)
してたのに (We used to be)
なぜ会えないの (Why can’t we meet each other?)

ᅳᄅ
ᄀ
ᇂᄀ
ᅥ
ᅦᄎ
ᆫᄒ
ᅵ
ᆻᄂ
ᅢ
ᆫᄃ
ᅳ
ᅦ (We were close before)
ᄋᄌ
ᅵ
ᆫᄋ
ᅦ
ᅡᅣ
ᄂ (and we’re not)
ᅳᄋ
ᄀ
ᅵᅲ
ᄋᄅ
ᆯᄋ
ᅳ
ᆯᄀ
ᅡ
ᅩᅡ
ᄑ (I want to know the reason why)

0.92,
0.80,
0.91

E(A2 ),
J(A2 ),
K(A2 )

Do you wanna build a snowman?
Or ride our bike around the halls?
I think some company is overdue
I’ve started talking to the pictures on the walls!

雪だるま作ろう (Let’s build a snowman)
自転車に乗ろう(Let’s ride a bike)
ずっと一人でいると (When I’m alone all the time)
壁の絵とおしゃべりしちゃう(I’m almost talking to the pictures on the walls)

ᇀᄋ
ᅡ
ᄀ
ᅵᄂ
ᆫᄉ
ᅮ
ᅡᄅ
ᆷᄆ
ᅡ
ᆫᄃ
ᅡ
ᅳ
ᆯᄅ
ᅢ? (Do you wanna build a snowman?)
이
ᅡ
ᄂᄆ
ᆫᄌ
ᅧ
ᅡᄌ
ᆫᄀ
ᅥ
ᅥᄐ
ᆯᄅ
ᅡ
ᅢ? (or do you wanna ride a bike?)
ᅵᅦ
ᄋ
ᄌᄂ
ᆫᄂ
ᅳ
ᅡᅩ
ᄃᄌ
ᅵᅧ
ᄎᄀ
ᅡᅡ
ᄂᄇ
ᅪ (Seems I’m getting tired)
ᆨᄋ
ᅧ
ᄇ
ᅦᅡ
ᄃᄆ
ᆯᄋ
ᅡ
ᆯᄒ
ᅳ
ᅡᅧ
ᄆᄂ
ᆯᄀ
ᅩ
ᅩᄋ
ᆻᄌ
ᅵ
ᅡ
ᆭᄋ
ᅡ
(because I’ve started talking to the walls)

0.79,
0.73,
0.82

E(B2 ),
J(B2 ),
K(B2 )

It gets a little lonely
All these empty rooms
Just watching the hours tick by

さびしい部屋で (In a lonely room)
柱時計 (the wall clock)
見てたりするの (I look at or something)

ᅡᄉ
ᄉ
ᆯᄋ
ᅵ
ᆫᄌ
ᅳ
ᅩᄀ
ᆷᄋ
ᅳ
ᅬᄅ
ᅩᄋ
ᅯ (In fact, I’m a little lonely)
ᄐᄇ
ᆼ
ᅥ
ᆫᄇ
ᅵ
ᆼᄋ
ᅡ
ᅦᄉ
ᆫ (In empty rooms)
ᅥ
ᅵᅨ
ᄉ
고
시
ᄅᄆ
ᆫᄃ
ᅡ
ᆯᄅ
ᅳ
ᅧ (All I can hear is the clock’s ticking)

0.90,
0.88,
0.96

Table 3. Lyric excerpt from “Do You Want to Build a Snowman” from the animation “Frozen,” singable in all languages.
Sections A1 and A2 form a musically similar pair, while B1 and B2 are also musically similar to each other. Each section isdenoted as E(A1 ), . . . , E(B2 ) in English, J(A1 ), . . . , J(B2 ) in Japanese, and K(A1 ), . . . , K(B2 ) in Korean.
4.1 Line Syllable Count Distance
It is crucial to preserve the syllable counts between theoriginal and translated lyrics for each line as similar as
possible in order to maintain the integrity of a song’s
melody [21]. Therefore, it is unsurprising that our evaluation framework incorporates a metric to assess the differences in syllable counts. Let the line syllable counts fora pair of lyrics that consist of n lines, X = {x1 , ..., xn }and X̃ = {x˜1 , ..., x˜n } be denoted as {syl(x1 ), ..., syl(xn )}and {syl(x˜1 ), ..., syl(x˜n )} where each element refers to
the syllable count of each line. For instance, if the firstline of the English lyrics X is “Silent night holy night”
and the corresponding line in the Korean lyrics X̃ is“Goyohanbam-georukhanbam (ᄀ
ᅩᅭ
ᄋᄒ
ᆫᄇ
ᅡ
ᆷᄀ
ᅡ
ᅥᄅ
ᆨᄒ
ᅮ
ᆫᄇ
ᅡ
ᆷ)”, theᅡvalue of syl(x1 ) is 6 and syl(x˜1 ) is 8. We define the linesyllable count distance between a pair of lyrics X andX̃ (Dissyl (X, X̃)) in order to evaluate the disparities insyllable counts, as follows.
1
Dissyl (X, X̃) = 2n

Pn

i=1 (

|syl(xi )−syl(x˜i )| |syl(xi )−syl(x˜i )|+)syl(xi )syl(x˜i )(1)
We compare the line syllable count distance of singable
and non-singable lyrics. As shown in Table 2, non-singablelyrics display a considerably greater Dissyl (X, X̃) compared to singable lyrics due to the varying syllable countrequirements across languages.
4.2 Phoneme Repetition Similarity
Rhyme, defined as the repetition of a vowel sound and any
subsequent sounds [22], has historically been a fundamental element in the realm of poetry, including in Western
languages like English. However, the concept of rhyme has
not been as prevalent in Japanese or Korean poetry [23].
In fact, traditional Korean poetry did not incorporate thisconcept [24]. Despite the increasing tendency to adopt theconcept of rhyme in Japanese and Korean lyrics due tointercultural exchanges, we observed that lyrics in theselanguages often rely more on repeating grammatical elements. For example, in section A1 of Table 3, the Japanesepair “tsukurou (作ろう, Let’s build)” and “asobou (遊ぼう,Let’s play)” generates a sense of rhyme because both end
776with the same conjugation “ou” meaning “let’s”. Similarly,
in Section A2 , the Korean pair “mandeullae (ᄆ
ᆫᅳ
ᅡ
ᆯ
ᄃᄅ
ᅢ, Do
you wanna build)” and “tallae (ᄐᆯ라ᅢ, Do you wanna ride)”creates a sense of repetition because both end with “llae”
meaning “Do you wanna”. Another example is the repetition of particles at the end of sentences, such as “yo (よ)”ᅭ)” and “da (다)” inand “no (の)” in Japanese and “yo (ᄋKorean, which convey cultural nuances related to formality.
We therefore propose that English, Japanese, and Korean
share common ground in adopting phoneme repetition for
poetic expression. However, as such repetition is not necessarily called rhyme in Japanese and Korean, we will refrainfrom using the term “rhyme” and instead employ the term“phoneme repetition.”
We noticed that each section’s degree of phoneme repetition remains consistent across different languages when
the lyrics are singable. For example, in Table 3, the firstsection of the original English lyrics (E(A1 )) displays astrong degree of phoneme repetition, with three rhyming
pairs: “come-come”, “play-away”, and “anymore-door” (Inthis paper, we denote a section as an uppercase with anumber and a line as a lower case with a number). Similarly, both the Japanese and Korean translations (J(A1 ),K(A1 )) also exhibit a substantial degree of phoneme repetition, featuring three pairs of repeated phonemes in each:“doa (ドア)”-“dou (どう)”, “tsukurou (作ろう)”-“asobou(遊ぼう)”, “akete (開けて)”-“shite (して)” in Japanese,ᇀ아ᅵ)”-“gachi (ᄀᇀ아ᅵ)”, “mandeul (ᄆ
ᆫᄃ
ᅡ
ᆯ)”ᅳ
and “gachi (ᄀ“eonnireul (ᄋᆫ너ᅵᄅᆯ)”, “mandeullae (ᄆ
ᅳ
ᆫᄃ
ᅡ
ᆯᄅ
ᅳ
ᅢ)”-“simsimhae
(ᄉ
ᆷᄉ
ᅵ
ᆷᄒ
ᅵ
ᅢ)” in Korean. However, we realized that it is notfair to directly compare the number of phoneme repetitions
when attempting to quantify the degree of phoneme repetition as each language has a different number of vowels
and consonants: English has 15 vowels and 24 consonants,
whereas Japanese has 5 and 15 and Korean has 21 and19. Hence, in an attempt to minimize the differences in
the number of phonemes, we treated acoustically similar
vowels as the same vowel in English, such as ‘IH’-‘IY’,‘UH’-‘UW’, or ‘EH’-‘AE’(e.g., ’mass’ and ’mess’) [25]because they can still form slant rhymes [26]. Conversely,
we considered ‘A’-‘YA’, ‘O’-‘YO’, and ‘U’-‘YU’ as sep-Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023
SourceTargetSingableNon-singable

EnglishJapanese
Korean

0.69
0.97

0.56
0.72

Japanese

EnglishKorean

0.79
0.80

0.61
0.48

KoreanEnglishJapanese

0.97
0.80

0.78
0.50

Table 4. The average phoneme repetition similarity(Simpho ) of singable lyrics and non-singable lyrics.
arate vowels in Japanese, as they are unlikely to function
as the same grammatical components. In Korean, we regarded the perceptually similar vowels (e.g., ‘AE’-‘E’or‘OE’-‘OI’-‘OAE’) as the same vowels [27, 28].
To quantitatively represent the degree of phoneme repetition, we utilized the concept of distinct-2, the ratio of
the number of distinct bi-grams to the total number of bigrams [29]. While the original concept formed bi-grams
using two consecutive words, we used two consecutive
phonemes to assess the degree of repetition because lower
distinct-2 values indicate higher repetition and vice versa.
The phoneme distinct-2 (pho) of a section Xi , is defined asfollows:
unique bi-gram # in X
pho(Xi ) = total bi-gram # in X i .
i(2)

For example, consider a section with a single line “twinkle
twinkle little star”, denoted as X1 . First, we decomposed
the section into phonemes and added the ‘<eos>’to eachline: ‘T’, ‘W’, ‘IH’, ‘NG’, ‘K’, ‘AH’, . . . , ‘S’, ‘T’, ‘AA’,‘R’, and ‘<eos>’. Next, we grouped each component into
bi-grams: ‘TW’, ‘WIH’, ‘IHNG’, ‘NGK’, ‘KAH’, ‘AHL’,. . . , ‘ST’, ‘TAA’, ‘AAR’, ‘R<eos>’. Finally, we calculated the phoneme distinct-2 of the section (pho(X1 )) by
dividing the number of unique bi-grams by the total number of bi-grams (17/23 = 0.74). To measure the similarity
between two sections in terms of the degree of phoneme
repetition, we introduce the phoneme repetition similarity (Simpho ). Given two sets of lyrics with m sections,
X = {X1 , ..., Xm } and X̃ = {X̃1 , ..., X˜m }, the phoneme
repetition similarity between X and X̃ is defined as theSpearman correlation between {pho(X1 ), ..., pho(Xm )}˜ )}, as shown below.
and {pho(X̃1 ), ..., pho(XmSimpho (X, X̃) = corr({pho(X1 ), ..., pho(Xm )}, {pho(X̃1 ), ..., pho(X˜m )})(3)
We present the statistical results for the average phoneme
repetition similarity of singable and non-singable lyrics inTable 4. The table clearly exhibits a higher correlation
between the original lyrics and singable translated lyrics interms of the phoneme distinct-2 than non-singable lyrics.
This result suggests that singable lyric translation takes
into account the degree of phoneme repetition within eachsection to convey a sense of repetition for that section.
4.3 Musical Structure DistanceUpon examining our section-divided singable lyrics data,
we identified two tendencies in lyrics when musical sections

Figure 1. Musical self-dissimilarity matrices for English,
Japanese, and Korean versions of the K-pop song “Icy” by
ITZY. Dissimilarity between the i-th and the j-th section
was computed using diss(Xi , Xj ).
are repeated (e.g., the repetition of the chorus). First, weobserved that musically similar sections tend to share thesame phonemes and, as expected, the same phrases. For
instance, in Table 3, musically similar sections share thesame vowels (e.g., “why” in E(B1 ) and “by” in E(B2 )) or
identical phrases (e.g., “Do you wanna build a snowman” inE(A1 ) and E(A2 )) in order to create a sense of consistency.
As a result, when calculating the phoneme distinct-2 (pho)for two concatenated sections, musically similar sections
are likely to have smaller values than musically different
sections. For example, pho(E(A1 ++ A2 )) is 0.70 (‘++’ denotes the concatenation of text), whereas pho(E(A1 ++B1 ))is 0.82, where A1 is musically similar to A2 but not to B1 .
However, a low value of pho does not always imply musicalsimilarity, as a meager pho value in one section could resultin a low pho of two concatenated sections despite the musical dissimilarity (e.g., “lalalalalalalalalalalalalala” ++ “doyou wanna build a snowman?”). From this case, we derived
our second observation that a significant difference in pho
for each section could imply musical differences. Accordingly, we also realized that musically similar sections tend
to have a similar degree of pho. For instance, in Table 3,
both A1 and A2 , a set of musically similar sections, exhibit
relatively low pho, indicating a strong degree of phoneme
repetition (rhyme), with similar values to each other acrossall languages. Likewise, both B1 and B2 , another pair ofmusically similar sections, demonstrate a higher pho, withsimilar values to each, in all languages.
Therefore, to quantify the musical similarity between
sections, we examined whether they have 1) a tendency toshare the same phoneme by obtaining pho(Xi ++Xj ), and2) similar pho values by calculating |pho(Xi ) − pho(Xj )|.
Given that higher values represent dissimilarity in bothcases, we define the musical dissimilarity between two
sections, diss(Xi , Xj ), as the sum of these two values, as
follows.
diss(Xi , Xj ) = pho(Xi ++Xj ) + |pho(Xi ) − pho(Xj )| (4)As shown in Figure 1, self-dissimilarity matrices employing our definition of musical dissimilarity look highly similar across English, Japanese, and Korean, where all are
singable, visually representing musical structure.
Finally, we quantitatively evaluated the distance between
matrices. We refer to this distance between matrices as themusical structure distance, as it represents the structural element of lyrics. In summary, the musical structure distance777

Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023
SourceTargetSingableNon-singable(Human)Non-singable(Machine)

EnglishJapanese
Korean

0.14
0.10

0.26
0.15

0.30
0.15

Japanese

EnglishKorean

0.13
0.11

0.20
0.13

0.18
0.14

KoreanEnglishJapanese

0.10
0.11

0.10
0.10

0.10
0.17

Table 5. The average musical structure distance (Dismus )of singable lyrics and non-singable lyrics.
between lyrics in different languages X and X̃, each consisting of m sections, Dismus (X, X̃), is defined as follows:
Dismus (X, X̃) = m12qP

m2i,j=1 (diss(Xi , Xj ) − diss(X̃i , X̃j ))(5)
In Table 5, we provide a summary of the average musicalstructure distance for singable lyrics, human-translated nonsingable lyrics, and machine-translated non-singable lyrics
generated by automatically translating official singablelyrics from 80 English, 162 Japanese, and 161 Korean
songs using Google Translator. Our findings show thatsingable lyrics exhibit the lowest Dismus values, whilemachine-translated non-singable lyrics display the highest, suggesting that machine-translated ones lack structural
coherence the most. As human-translated non-singablelyrics maintain structural coherence in aspects such as word
choice and nuances, they demonstrate lower distances thanmachine-translated counterparts.
5. EVALUATING SEMANTICS
Semantic relatedness to the original lyrics is by no means
less fundamental than syllable counts, phoneme repetition,and structural factors [18]. We therefore introduce a fourth
metric, semantic similarity, to ensure the semantic relevanceof translated lyrics to the original.
5.1 Semantic Similarity
To numerically assess the semantic textual similarity (sts)between a pair of lyrics, we first obtained the contextual
embeddings of each text from lyrics using a pre-trained
Sentence BERT model 1 [31] and then calculated the cosine
similarity between the embeddings. As this model wastrained for English, the Japanese and Korean lyrics wereautomatically translated using Google Translator before
obtaining the embeddings.
We started by examining hierarchical semantic similarity
using cross-scape plots [32], as shown in Figure 2. Givena pair of lyrics X = {x1 , . . . , xn } and X̃ = {x˜1 , . . . , x˜n }with n lines each, the first (leftmost) block of the lowest
line represents the semantic textual similarity between x1
and x˜1 (denoted as sts(x1 , x˜1 )) while the last (rightmost)block signifies sts(xn , x˜n ). The first (leftmost) block of thesecond-lowest line denotes sts(x1 ++x2 , x˜1 ++x˜2 ), and thesecond block corresponds to sts(x2 ++x3 , x˜2 ++x˜3 ). Lastly,
1 We used all-MiniLM-L6-v2 [30].

778

Figure 2. Semantic similarity cross-scape plot for the J-pop
song “A Thousand Winds” between English and Japanese(Left), English and Korean (Middle), and Japanese andKorean (Right). Any value less than 0 was regarded as 0.
Line#

EnglishJapanese(English translation)sts

1

please do not stand at
my grave and weep.

私のお墓の前で(in front of my grave)

0.56

2

I am not there, I do not sleep

泣かないでください(please stop crying)

0.27

1,
2

please do not stand at
my grave and weep. I am
not there, I do not sleep

私のお墓の前で泣かないでください(in front of my grave.
please stop crying.)

0.76

Table 6. Semantic textual similarity (sts) between English
and Japanese versions of “A Thousand Winds”.
the highest block (line) represents the similarity between
the entire lyrics, sts(x1 ++ · · · ++ xn , x˜1 ++ · · · ++ x˜n ).
In each plot of Figure 2, there are semantic disparities
at lower levels, but similarities increase at higher (broader)levels. We have two explanations for this. First, the numberof musical notes within a single lyric line may be adequate
to deliver a specific message in one language but insufficientin another language. Therefore, it is common for a message conveyed in one line in one language to span two lines
in another language. As an example, we provide Table 6,
which presents the semantic textual similarity (sts) between
Japanese and English lyrics of the J-pop song “A Thousand
Winds (千の風になって)”. As demonstrated in the table,
the similarity between Japanese and English at a broader
level (sts(x1 ++x2 , x˜1 ++x˜2 )) can be higher than at the line
level (sts(x1 , x˜1 ), sts(x2 , x˜2 )) because Japanese generally
requires more syllables than English and it often takes twolines in Japanese to express a single-line message in English. Second, the semantic similarities at broader levels
can be higher because of grammatical/linguistic differences.
Each language has its own natural word order patterns. For
example, in the phrase “I’m going to travel to find the gold,”
it is natural in English to mention “I’m going to travel” before “to find the gold.” However, in Japanese and Korean,
expressing “to find the gold (金を探しに, ᄀ
ᆷᅳ
ᅳ
ᆯ
ᄋᄎ
ᆽᄋ
ᅡ
ᅳᄅ
ᅥ)”
before “I’m going to travel (旅に出る, 떠ᄂᆫ다ᅡ)” is a more
typical and natural construction. Table 7 shows that thesedifferences between languages make line-level semantic assessments insufficient. Since lines 1, 2, and 3 in the English
version correspond to lines 3, 1, and 2 respectively in theJapanese version, these pairs exhibit low semantic similarities at the line level (sts(x1 , x˜1 ), sts(x2 , x˜2 ), sts(x3 , x˜3 )),while demonstrating higher similarity when considered as awhole (sts(x1 ++x2 ++x3 , x˜1 ++x˜2 ++x3 )).
Considering these factors, it becomes evident that

Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023
Line#

EnglishJapanese(English translation)

sts

1

Dare to try and reach out
for hеaven

望むように生きるなら (If you want
to become what you’re meant to be)

0.22

2

You must become
what you’rе meant to be

星からの金を求め(to find the gold from stars)

0.27

3

And bring the gold ofheaven to the world

一人旅に出るのよ(Dare to embark on a solo journey)

0.13

1-3

Dare to try and reach out
for hеaven You must become
what you’rе meant to be
And bring the gold ofheaven to the world

望むように生きるなら星からの金を求め一人旅に出るのよ(Dare to embark on a solo journey
0.53
if you want to become what you’remeant to be to find the gold from stars)

Table 7. Semantic textual similarity (sts) between English
and Japanese versions of “Gold von den Sternen”.

SingableNon-singable

SourceTargetline

sectionline

sectionEnglishJapanese
Korean

0.40
0.42

0.54
0.55

0.64
0.70

0.74
0.76

Japanese

EnglishKorean

0.47
0.52

0.59
0.61

0.66
0.77

0.72
0.79

KoreanEnglishJapanese

0.53
0.52

0.63
0.61

0.78
0.73

0.81
0.75

Table 8. The average line-wise semantic similarity andsection-wise semantic similarity (Simsem ) of singable andnon-singable lyrics.
6. DISCUSSIONS AND CONCLUSIONS

Figure 3. The line-wise (Left) and section-wise (Right)
semantic similarity matrices between Japanese and Korean
versions of “Wie wird man seinen Schatten los?”singable lyric translations do not prioritize line-wise semantic similarity. Rather, we observed that singable translations
aim to preserve semantic connections at the section level
since the organization of the lyric storyline follows a sectionwise approach. To illustrate this, we present Figure 3, whichdisplays both line-wise and section-wise semantic similarity
matrices for the Japanese and Korean versions of “How doyou get rid of your shadow? (Wie wird man seinen Schatten
los?)” from the German musical “Mozart!”. As shown in
the Figure, the section-wise matrix represents the semantic
relatedness more clearly than the line-wise matrix.
Therefore, we propose assessing section-wise semantic
relatedness for evaluating singable lyric translation. To
achieve this, we define the semantic similarity between apair of lyrics X = {X1 , ..., Xm } and X̃ = {X̃1 , ..., X˜m },consisting of m sections and n = n(X1 ) + · · · + n(Xm )lines, where n(Xi ) denotes the number of lines in the i-th
section, as follows:
Simsem (X, X̃) =

Pm

i=1 (

n(Xi )
n sts(Xi , X̃i )).

(6)

Table 8 compares singable and non-singable
Pnlyrics in termsof line-wise semantic similarities ( n1 i=1 sts(xi , x̃i ))and section-wise similarities, using our proposed metric(Simsem ). The table reveals that non-singable translations exhibit high semantic similarity for both line-wiseand section-wise measures, with similar values for each. In
contrast, singable translations show low line-wise similarity,
as expected, since they do not prioritize line-wise semantic
similarity. However, when evaluated using section-wisesimilarity, they display a level of similarity comparable tothat between “Machine learning is so easy” and “Deep learning is so straightforward”, which is 0.623 when measured
with the same pre-trained model [30].

In this paper, we introduced a computational evaluation
framework for singable lyric translation, grounded in themusical, linguistic, and cultural understanding of lyrics,
comprised of four evaluation metrics, line syllable countdistance (Dissyl ), phoneme repetition similarity (Simpho ),musical structure distance (Dismus ), and semantic similarity (Simsem ). These metrics are designed to ensure that thetranslated lyrics maintain the integrity of melodies, degreeof phoneme repetition, structural coherence, and semantics of the original lyrics. Our framework is automated,guaranteeing objectivity and efficiency in terms of time andcost. We showed the efficacy of our evaluation metrics byoffering comparative statistics between singable and nonsingable lyrics. In addition, our analysis revealed that thedegree of phoneme repetition in the original lyrics is frequently mirrored in the translated lyrics, musically similar
sections tend to share the same phonemes and display comparable degrees of phoneme repetition, and section-wise
analysis is better suited for evaluating semantic similarity
for lyric translation than line-wise analysis.
Nonetheless, there remains room for improvement. Although we have assembled a singable lyrics dataset, aligned
across English, Japanese, and Korean, our dataset has some
limitations; it lacks musical information and its volume islimited. As a result, we have not been able to incorporate
musical notes into our experiment or conduct comparative
studies across various genres. We recognize that an ideal
lyric translation evaluation system should take into accountthe relationship between musical notes and phonemes, aswell as adapt to different genres. Moreover, although wehave endeavored to incorporate cultural understandings ofpoetry in different languages, we acknowledge the need fordeeper cultural considerations. For example, we noticed
that cultural similarities might have an impact on the extent
of semantic similarities. This is demonstrated in an English
translation of “MIC Drop”, a K-pop song by BTS originally
written in Korean, made by YouTuber Iris Phuong. The
translated singable lyrics do not include a translation of theterm “hyodo (효ᅩᄃ, taking care of parents)” as there is no direct equivalent in English, while the Japanese version of thesong effortlessly conveys the concept as “koukou (孝行)”.
In the future, we aim to expand our dataset to contribute
more to lyric translation studies and to further explore theimpact of genre and cultural influences on lyric translation.
779

Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023

7. ACKNOWLEDGMENTS
This work was supported in part by JST CREST Grant
Number JPMJCR20D4.
8. REFERENCES[1] M. Mateo, “Music and translation,” Handbook of translation studies, vol. 3, pp. 115–121, 2012.
[2] Ş. Susam-Sarajeva, “Translation and music: Changing
perspectives, frameworks and significance,” The Translator, vol. 14, no. 2, pp. 187–200, 2008.

[13] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and
Y. Artzi, “BERTScore: Evaluating text generation withbert,” in International Conference on Learning Representations (ICLR), 2019.
[14] I. Marc, “Travelling songs: On popular music transfer
and translation,” IASPM Journal, vol. 5, no. 2, pp. 3–21,
2015.
[15] E. C. Hui-tung, “Translation of songs,” An Encyclopediaof Practical Translation and Interpreting, p. 351, 2019.

[4] P. Low, “Translating songs that rhyme,” Perspectives:
Studies in translatology, vol. 16, no. 1-2, pp. 1–20,
2008.

[16] Y. Yang, Y. Zhang, C. Tar, and J. Baldridge, “PAWSX: A cross-lingual adversarial dataset for paraphrase
identification,” in Proceedings of the 2019 Conferenceon Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp.
3687–3692.

[5] ——, “The pentathlon approach to translating songs,”
in Song and significance. Brill, 2005, pp. 185–212.

[17] H. S. Drinker, “On translating vocal texts,” The Musical
Quarterly, vol. 36, no. 2, pp. 225–240, 1950.

[6] F. Guo, C. Zhang, Z. Zhang, Q. He, K. Zhang, J. Xie,and J. Boyd-Graber, “Automatic song translation for
tonal languages,” in Findings of the Association for
Computational Linguistics (ACL). Dublin, Ireland:
Association for Computational Linguistics, May
2022, pp. 729–743. [Online]. Available: https://aclanthology.org/2022.findings-acl.60[18] J. Franzon, “Three dimensions of singability. An approach to subtitled and sung translations,” Text andTune. On the Association of Music and Lyrics in Sung
Verse. Bern: Peter Lang, pp. 333–346, 2015.

[3] S. Spaeth, “Translating to music,” The Musical Quarterly, vol. 1, no. 2, pp. 291–298, 1915.

[7] L. Ou, X. Ma, M.-Y. Kan, and Y. Wang, “Songs acrossborders: Singable and controllable neural lyric translation,” arXiv preprint arXiv:2305.16816, 2023.

[19] J. P. G. Mahedero, A. MartÍnez, P. Cano, M. Koppenberger, and F. Gouyon, “Natural language processing oflyrics,” in Proceedings of the 13th Annual ACM International Conference on Multimedia, ser. MULTIMEDIA’05. Association for Computing Machinery, 2005, pp.
475–478.

[8] C. Li, K. Fan, J. Bu, B. Chen, Z. Huang, and Z. Yu,“Translate the beauty in songs: Jointly learning toalign melody and translate lyrics,” arXiv preprint
arXiv:2303.15705, 2023.

[20] K. Watanabe and M. Goto, “A chorus-section detection
method for lyrics text.” in Proceedings of the 21th International Conference on Music Information Retrieval(ISMIR), 2020, pp. 351–359.

[9] K. Watanabe and M. Goto, “Lyrics information processing: Analysis, generation, and applications,” in Proceedings of the 1st Workshop on NLP for Music andAudio (NLP4MusA), 2020, pp. 6–12.

[21] R. Apter and M. Herman, “Translating art songs for
performance: Rachmaninoff’s six choral songs,” Translation Review, vol. 84, no. 1, pp. 27–42, 2012.

[10] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“BLEU: A method for automatic evaluation of machinetranslation,” in Proceedings of the 40th annual meetingof the Association for Computational Linguistics (ACL),
2002, pp. 311–318.
[11] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text summarization branches
out, 2004, pp. 74–81.
[12] S. Banerjee and A. Lavie, “METEOR: An automatic
metric for MT evaluation with improved correlation
with human judgments,” in Proceedings of the ACL
Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,
2005.
780

[22] A. Bain, “English composition and rhetoric,(2nd american ed.),” New York: D. Appleton and Company, 1867.
[23] N. Manabe, “Globalization and Japanese creativity:
Adaptations of Japanese language to rap,” Ethnomusicology, vol. 50, no. 1, pp. 1–36, 2006.
[24] Y. B. Yoon and B. L. Derwing, “A language without
a rhyme: Syllable structure experiments in Korean,”Canadian Journal of Linguistics/Revue canadienne delinguistique, vol. 46, no. 3-4, pp. 187–237, 2001.
[25] A. K. Syrdal and H. S. Gopal, “A perceptual model ofvowel recognition based on the auditory representationof American English vowels,” The Journal of the Acoustical Society of America, vol. 79, no. 4, pp. 1086–1100,
1986.

Proceedings of the 24th ISMIR Conference, Milan, Italy, November 5-9, 2023

[26] K. Hanson, “Formal variation in the rhymes of RobertPinsky’s the inferno of dante,” Language and Literature,
vol. 12, no. 4, pp. 309–337, 2003.
[27] C. Ito, Y. Kang, and M. Kenstowicz, “The adaptation ofJapanese loanwords into Korean,” MIT Working Papers
in Linguistics, vol. 52, no. 2006, pp. 65–104, 2006.
[28] S.-E. Chang, “Enhancement effects of clear speech andword-initial position in Korean glides.” The Journal of
the Acoustical Society of America, 2017.
[29] J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan,“A diversity-promoting objective function for neural conversation models,” in Proceedings of the 2016 Conference of the North American Chapter of the Association
for Computational Linguistics (NACCL) : Human Language Technologies, 2016, pp. 110–119.
[30] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and
M. Zhou, “MiniLM: Deep self-attention distillation fortask-agnostic compression of pre-trained transformers,”
Advances in Neural Information Processing Systems(NeurIPS), vol. 33, pp. 5776–5788, 2020.
[31] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992.
[32] S. Park, T. Kwon, J. Lee, J. Kim, and J. Nam, “Across-scape plot representation for visualizing symbolic
melodic similarity,” in Proceedings of the 20th International Conference on Music Information Retrieval(ISMIR), 2019, pp. 423–430.

781