言語処理学会 第30回年次大会 発表論文集（2024年3月）

Creating Heterogenous Transcription of English and Japanese ona Multilingual Audio File
Yuika Sun
Los Altos High School, CA 94022, USA
yuhyeung@gmail.com

Code-switching (CS) is the switching between
multiple languages in speech. Current literature oncode-switching transcription technology has been

However, when the user code-switches from English andJapanese, the closed captions generate incomprehensible
characters. A similar phenomenon is observed in
OpenAI’s Whisper. After the first 30 seconds of theaudio file, the transcription outputs romaji for Japanesesuccessful in translating the CS transcripts into one
language but there yet to exist satisfactory technology toaccurately transcribe one single speaker’s CS audio intoCS transcripts. Current speech-to-text (STT) technology,
although produces accurate transcriptions, is limited tomonolingual sentences. In this paper, the focus is on an

characters if the speaker code-switches from English toJapanese. If a language like Japanese is specified using aparameter from the beginning, it will completelytranslate the English sentence into Japanese, regardlessof whether the speaker has code-switched or not.
However, if the speaker code-switches within the first 30

unexplored area: expanding STT to English and Japanese
CS language transcription without utilizing translation
software.

Bilingualism, the ability to speak two languages, isseconds of the audio file, the software accuratelytranscribes the Japanese portion of the code-switchedaudio with the appropriate katakana, kanji, and hiragana,and the English portion of the code-switched audio withcorrect English words.
Another technology that transcribes spoken

prevalent—more than half of the world is bilingual. [1]This leads many to code-switch, a phenomenon where
one speaks multiple languages interchangeably in spoken
sentences.
Accurately transcribing code-switched audio isdifficult, as not many software can successfully

language is speech diarization, which involves
determining "who spoke when", distinguishing between
speech and non-speech sounds, and marking speakerchanges within the identified speech. [2] However,speech diarization could not be used for this specificissue of recognizing code-switched language spoken bytranscribe a multilingual audio file into heterogenous text
correctly.
A program using Heterogenous Language Detection
and Re-transcription (HLDR) 1 for multilingual audio
files with Japanese and English code-switching wasdeveloped to address existing gaps in functionality of

one single person. Speech diarization requires multiple
distinct acoustic features to identify the two borders
between CS and non-CS speech. This paper focuses on
accurately transcribing CS from one speaker withuniform acoustic features in their speech, making speech
diarization an unsuitable technology.

current STT software. There exists some audio-to-texttranscription software that resolves constituents of thisproblem—the task of transcribing one language at atime.
For instance, Adobe Premiere Pro’s automatictranscription feature immediately auto-transcribes audio.

Multilingualtranscription
technology
faces
challenges in accuracy due to existing technologies’ sole
focus on monolingual transcription. [3] Creating STT
software with the capability of handling code-switching
between English and Japanese will make video subtitles
more accessible to more people, reducing the time for

Abstract

1 Introduction

1

video

editing

andtranscription

forthose

https://github.com/LearnML-Me/HLDR/― 1264 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

whocode-switch between English and Japanese in theirvideos. In addition, creating a STT will open more doors
in the world of STT for East Asian languages. For
instance, Japanese, Chinese, Korean, and English havemutually borrowed loanwords,transcription. Two English transcriptions are used to findthe commas between each phrase. To detect the noise, it
is understood that the English translation has a relatively
low word error rate (WER). WER is the ratio ofsubstitutions (S), deletions (D), and insertions (I) ofwords in a transcript to the overall number of spoken orFigure 1
allowing forincreased efficiency in creating these language models.

intended words (N1).

2 MethodFigure 2

In the realm of linguistic analysis, a novel concept,[4] Considering this, the two English transcriptions are"noise" is introduced within the context ofcode-switching for this paper. Unlike the conventional
understanding of noise as auditory interference, in thiscontext, it denotes the unintended linguistic distortions
that arise during the transcription of code-switched
conversations. In audio files with code-switchedhighly likely to be very similar due to its WER.
Therefore, a phrase that has a high WER whencomparing the two English transcription files will bedetected as “noise”. In HLDR approach, proper
capitalization is ignored. Similarly, the same HLDR
principle may be used with the Japanese STT software,

language, noise emerges as an inadvertent byproduct.
This distortion materializes when transcription software
encounters code-switching instances, unable todistinguish between the original language and theinterjected foreign language. Consequently, thetranscribed output consists of unfamiliar and seemingly

but an appropriate Japanese STT software other thandictation-kit was not found with a satisfactory WER asof the writing of this paper.
Materials: Julius [5], dictation-kit [6], DeepSpeech [7],running on Ubuntu 22.04 LTS, an audio recording
containing both English and Japanese sentences.

nonsensical words in the original language, illustrating
the challenges inherent in accurately capturing thecomplexity of code-switched language. This paper aims
to eliminate “noise” from transcripts that deal withcode-switched language. “noise” is taken out by adistinction of where to start the and stop a transcription
in one language and append the transcription in thesecond language. To accomplish this task, two English
language models and one Japanese model are utilized tofind the noise relationship between them and merge theminto an English and Japanese heterogenous final

3 Results
In this trial a heterogenous code-switched transcription
was tested.
Original intended audio: “the first one, 全部 食べ ても 問題 ない, I have no problem eating all of this.”
Phase I:This phase is to get the raw speech-to-text (STT) textfile.
Transcriptions were made through neural networks.
Types include Deep Learning Neural Networks (DNN),― 1265 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Recurrent Neural Network (RNN), and Convolutional
neural network (CNN).
A difference between the Julius model and DeepSpeechmodel transcripts Julius transcribed it as “I have noproblem eating all this” while DeepSpeech transcribedThis phase is the key portion of this article. A Python
script is used to process 3 raw transcription files. The
basic functionality is like the following:••

Read 3 files and store them as 3 variables
Tokenize the strings by converting to string lists

the audio as “I have no problem eating all of this”. To
mitigate this concern, WER was used; there was one
word that caused the discrepancy: “of”, with a WER of12% when compared to the full phrase. When comparedto WER in other contexts outside of this paper, such as
the WER for Wit (25.87%), a free speech-to-text•••

Scan and record the location of comma “,”Normalize the strings by removing comma(s)Use diff Python library to get the difference of
EN1 and EN2 tokenized strings•Pick up the common words into a new list andidentify the English noise section.

software. [8] HLDR was used on the third portions of
EN1 and EN2 respectively to produce the final output,
which is “I have no problem eating all this”. To
intelligently choose the most accurate final output basedon nuances and semantics of each word using advanced
NLP libraries like spaCy and MeCab is left to future•Skip the same amount of noise from theJapanese result and inject the correct Japanesesentence into the English noise section.

•Restore the commas in their recorded locations(from step 3) in the combined string.

•Combine the string list to get the combinedstring.

research as improvement.
The existing ASR language models were leveraged toprocess the audio file. On 3 platforms: dictation-kit,
English and DeepSpeech, after running the processor
binary by specifying a language model, the essential
outcome of the recognized speech text was extracted
from the result with the combination of Linux commands“grep” and “sed”, the preliminary file JA.txt, EN1.txt,
EN2.txt are created respectively. The content is like thefollowing without quotation mark:•Japanese STT: dictation-kit (JA.txt)“と 過ごす と 、 全部 食べ て も 問題 ない 、 間の プロ 入り の です 。
”
• English STT: Julius (EN1.txt)“
the first one, timber habitable one nine nine, I have noproblem eating all this”•English STT: DeepSpeech (EN2.txt)“
the first one dim with habit demon the knight I have noproblem eating all of this”
Phase IIIf I section out the outputs from dictation-kit,DeepSpeech, and Julius into respective Python lists, Ican compare their phrases with each other. (Figure 3)

Figure 3
Each list has three items. Focusing on EN1 and EN2,
each word inside the first and third items are nearly
identical. Conversely, all the words inside of the second
item in the list are different. The second item is theportion in which the Japanese was being spoken. This
scenario is visualized in Figure 1. In this case, I utilize
this pattern in hopes of writing a code that substitutes theinconsistent portion of item 2 from transcription. The
code that could implement this is referenced in Figure 3.

4 Discussion
There exist several limitations to the method used in thisresearch paper. First, because this method can be
regarded as meta transcription, its quality is variable
based on the underlying transcription software’s quality
and language model training hours. Its accuracy also
depends on the field of the terminology between thespoken words/sentences and the trained model. The
WER must be at a minimum.
In the course of time, this problem will gradually be― 1266 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

improved and solved, but ultimately it needs a different
approach to address the issue at its foundational level. In
addition, the quality of the output when utilizing thispaper’s approach varies heavily on the spoken words andsentences. Correct Japanese punctuation in thetranscription such as commas and periods areinconsistent, which will cause the script to havedifficulty finding the border of the correct Japanesewords and noise coming from English. This can be
attributed to the fact that only one Japanese language
model has been identified that meets the specified

English code-switched sentences. Due to the loanwords
that exist between Japanese and Chinese, this method
would be most practical. By extension, it is worth
considering engineering a program that can process
low-resource languages.

5 Conclusion
In this paper, a strategy is introduced to merge English
and Japanese transcription software together using two
English and one Japanese transcription software and aPython script to find appropriate words to merge themrequirements, which include accurate punctuation and afully functional translation encompassing kanji, katakana,and hiragana. HLDR could be further improved byunderstanding instances when Japanese kanji is written
as hiragana and vice versa, and add the functionality ofallowing the user to choose between options.

together. Any combination of STT software, whether
commercial or open source, is the user’s choice and may
be used in the HLDR method, opening the door tometa-transcription. Similar to a technology term,
Redundant Array of Independent Disks (RAID), which
use multiple disks and combine them together to get a

There also exist several directions for future research on
this topic. First is expanding the English transcription
service from 2 to many, to get the best accuracy ofEnglish transcription. In addition, expanding theJapanese transcription service from 1 to many isimperative to get the most accurate insertion of Japanesebetter result in terms of speed and availability, [9] ahigher number of STT used in the HLDR method ispredicted to increase the accuracy and lower the cost of
the combined result.

code-switched transcription. Furthermore, expanding thismethod of accurate transcription of code-switchedspeech requires software libraries spaCy, NLTK, andMeCab for advanced NLP tasks such as semanticanalysis. The current strategy in this paper can onlytranscribe code-switched phrases, not individual words.

[1] Ana Inés Ansaldo, Karine Marcotte, Lilian Scherer,and Gaelle Raboyeau. Language therapy and bilingualaphasia: Clinical implications of psycholinguistic andneuroimaging research. In Journal of Neurolinguistics,Vol. 21, No.6, pp. 1, 2008.
[2] –. ScienceDirect. (Online) (Access Date: 1/3/24.)

Semantic analysis will allow for accurate insertion ofspecific words in text by considering the context of thesurrounding sentence. Another limitation is that of thescope of the transcription ability. This paper’s approachis adequate at transcribing code-switched speech with
one sentence is English, and one is Japanese, etc.

https://www.sciencedirect.com/topics/computer-science/
speaker-diarization.
[3] –. Machine Learning Research. (Online) (Access
Date: 1/3/24.)https://machinelearning.apple.com/research/towards-real-world.

However, tasks such as transcribing a spoken sentence
containing mixed words instead of mixed phrases
between English and Japanese is beyond the capabilitiesof the software. Finding a solution to this scenario would
require deep analysis of the transcription using advanced
NLP processing software such as spaCy or NLTK.

[4] –. ResearchGate. (Online) (Access Date: 1/3/24.)https://www.researchgate.net/publication/221488965_An_Empirical_Analysis_of_Word_Error_Rate_and_Keywo
rd_Error_Rate.
[5] A. Lee and T. Kawahara: Julius v4.5 (Online)(Access Date: 1/3/24) https://doi.org/10.5281/zenodo.25

Another direction is to statistically adopt the besttranslation of Japanese. The method would be morediverse if it included more languages, such asrecognizing, parsing, and transcribing Chinese and30395.
[6] –. Github dictation-kit. (Online) (Access Date:1/3/24) https://github.com/julius-speech/dictation-kit.

References― 1267 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

[7] Awni Hannun, Carl Case, Jared Casper, Bryan
Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger,
Sanjeev Satheesh, Shubho Sengupta, Adam Coates, andAndrew Y. Ng Deep Speech: Scaling up end-to-end
speech recognition arXiv preprint arXiv:1412.5567v2,2014.
[8] –. National Library of Medicine. (Online) (AccessDate: 1/6/24.)https://www.ncbi.nlm.nih.gov/pmc/articles/PMC725640
3/.
[9] Christian Zoubek, Sabine Seufert, and Andreas
Dewald. Generic RAID reassembly using block-level
entropy. In Digital Investigation, Vol. 16, pp. 1, 2016.

― 1268 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).