# Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations
## 1 Introduction  
Automatic speech recognition (ASR) is attracting a great deal of attention from the research community [1; 2; 3] for its diverse real-world applications. Although automatic lyric transcription (ALT) [4] has languished in comparison, it is still being applied to lyric alignment [5], query by singing [6], audio indexing [7], and music subtitling [8]. While both involve extracting linguistic content from a human voice stream, a singing voice implicates significant variations in pitch, duration, and intensity to a greater degree than speech. Thus, well-designed ASR models [4; 9] impair ALT performance, although they could be employed as a substitute for ALT. Notably, most ALT research in this area is primarily focused on English corpora [4], although there is some research [9] on Korean or other non-English language corpora [10] as well. One reason for this may be the lack of temporal alignment annotations for songs of those languages. Even when using CTC (Connectionist Temporal Classification) Loss [11], which allows model training in the absence of alignment information, some form of coarse alignment annotation is still necessary during data preprocessing when songs are segmented into shorter segments. To overcome the data challenge and the development lag in non-English ALT, it is natural to consider transferring priors from the more advanced English research ecosystem. To ensure high general cross-lingual priors rather than language-specific knowledge, self-supervised learning (SSL) can be a choice. Without annotated labels, SSL could better capture the general representations embedded in the language. In general, labels in the field of speech can vary, ranging from words to phonemes, so the learning relying solely on specific labels, such as words, could limit a model's generalization performance. However, SSL with phonemes circumvents this limitation by reducing dependence on specific labels, affording SSL models, such as HuBERT [12], exceptional generalization capabilities. In short, SSL can handle a wide array of speech data types, without being constrained by a single label. This versatility enables SSL models to better adapt to and address complex and evolving tasks in the realm of speech processing. Consequently, their generalization performance and application are potentially superior to that of supervised learning with specific labels [13]. This advantage broadens the scope of SSL and permits more adaptable and efficient solutions for speech processing. In theory, a pre-trained ASR model trained on an English corpus could be utilized to perform ALT from Korean singing voices. However, in general, the abrupt adaptation from priors of HuBERT-based SSL to target the Korean singing domain with a tiny dataset is not adequate. This is empirically shown in the experiment section. Additionally, the objective of ALT is to generate a word stream as output directly from the model. Alternatively, the model's output can initially produce a phoneme stream, which can then be converted into a word stream through a post-processing step. This approach is less dependent on a specific language (English), and enables the efficient utilization of priors for domain shift (to Korean). Here, the phoneme recognition in singing voices can serve as the initial phase of ALT. This paper aims to propose a method for Korean phoneme recognition based on singing voices, which implies that an additional post-processing step is necessary for word stream output. Figure 1 shows a flow diagram of the proposed method to build a Korean phoneme recognition model from Korean songs. In the first stage, the SSL model was trained to capture the general speech representation with 960 h of a large-scale English corpus. The pre-trained SSL model then adapts to a relatively small-scale 180 h of a Korean speech corpus. In that stage, the Korean phonemes were interpreted as similar to English ones. Finally, the Korean speech model was again adapted to the Korean song domain using 2.5 h of a tiny singing voice corpus.