CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units (2024)

Abstract: Multilingual code-switching research is often hindered by the lack and linguistically biased status of available datasets. To expand language representation, we synthesize code-switching data by replacing intonation units detected through PSST, a speech segmentation model fine-tuned from OpenAI’s Whisper, using a speech-to-text translation dataset, CoVoST 2. With our dataset, CoVoSwitch, spanning 13 languages, we evaluate the code-switching translation performance of two multilingual translation models, M2M-100 418M and NLLB-200 600M. We reveal that the inclusion of codeswitching units results in higher translation performance than monolingual settings and that models are better at code-switching translation into English than non-English. Further, low-resource languages gain most from integration of code-switched units when translating into English but much less when translating into non-English. Translations into lowresource languages also perform worse than even raw code-switched inputs. We find that systems excel at copying English tokens but struggle with non-English tokens, that the offtarget problem in monolingual settings is also relevant in code-switching settings, and that models hallucinate in code-switching translation by introducing words absent in both of the original source sentences. CoVoSwitch and code are available at https://github.com/ sophiayk20/covoswitch.1

heading: 1 Introduction
text: Code-switching (CSW), otherwise known as codemixing, refers to the use of linguistic units from multiple languages in a conversation or utterance (Pratapa et al., 2018). In general, researching codeswitching comprehensively is a complicated task due to the lack of code-switched data. One solution is to use existing code-switching datasets
1CoVoSwitch is released as a HuggingFace dataset. https: //huggingface.co/datasets/sophiayk20/covoswitch.
(Weller et al., 2022; Nguyen et al., 2023), but there is a limited number of such datasets and using them constrains research to the few language pairs that datasets are concentrated in, such as Spanish-English or Hindi-English (Winata et al., 2023). To alleviate the problem, previous work (Alastruey et al., 2023) brought together multiple datasets, such as Fisher (Cieri et al., 2004) and Bangor Miami (Deuchar et al., 2014). Nevertheless, in the multilingual setting, collecting data from multiple sources mixes different degrees of codeswitching and blocks parallel understanding across languages.
Alternatively, most works have introduced synthetic datasets (Winata et al., 2023). These have been based on linguistic theories, such as the Matrix Language Frame (MLF) Model (MyersScotton, 1997) and the Equivalence Constraint (Poplack, 1980). Applying the Equivalence Constraint requires the use of constituency parsers. (Rizvi et al., 2021) utilized the Stanford Parser (Klein and Manning, 2003) and the Berkeley Neural Parser (Kitaev and Klein, 2018; Kitaev et al., 2019). However, as of now, the Stanford Parser supports Arabic, Chinese, English, French, German, and Spanish, while the Berkeley Neural Parser supports Arabic, Basque, English, French, German, Hebrew, Hungarian, Korean, Polish, and Swedish. This presents a bottleneck in the number of languages that can be used for research and impedes the creation of code-switching data for unsupported or low-resource languages such as Tamil.
Synthetic datasets have also introduced codeswitching mainly based on words. These include random replacements based on words (Rijhwani et al., 2017; Xu and Yvon, 2021; Rizvi et al., 2021; Tarunesh et al., 2021) and replacements based on connected components of aligned words (Iyer et al., 2023). However, word-based switching may not completely reflect the code-switching phenomenon. Recent research (Pattichis et al., 2023) demon-
469
strated that code-switching is more common across intonation units than within as a result of looser syntactic relationships and that intonation units should therefore serve as new replacement units instead of words. This constraint is referred to as the Intonation Unit Boundary Constraint.
To expand language representation, experiment with intonation units as basis units of codeswitching, and reflect both linguistic and prosodic constraints, we synthesize data by following the Matrix Language Frame Model and the Intonation Unit Boundary Constraint. We keep English as the matrix language and embed segments from nonEnglish languages by replacing English intonation units of utterances from CoVoST 2 (Wang et al., 2021), a speech-to-text translation (S2TT) dataset, detected with PSST (Roll et al., 2023), an English prosodic speech segmentation model fine-tuned from OpenAI’s speech recognition model Whisper (Radford et al., 2023). Utilizing S2TT datasets is advantageous for several reasons. First, they include transcripts for both languages and audio files for one language in each pair, which allows the simultaneous incorporation of text and speech features in code-switching data creation. Moreover, recent datasets cover a multitude of high-resource and low-resource languages, which enables the inclusion of diverse language pairs for synthetic codeswitching data.
Meanwhile, we observe that while recent works (Zhang et al., 2023; Khatri et al., 2023) have demonstrated the translation performance of multilingual large language models with billions of parameters such as XGLM-7.5B and BLOOMZ-7b1 on codeswitching data, performance of multilingual neural machine translation (MNMT) models with millions of parameters remains relatively underexplored. We therefore measure the zero-shot code-switching translation performance of M2M-100 418M (Fan et al., 2021) and NLLB-200 600M (Costa-jussà et al., 2022), capable of multilingual translation for 100 and 200 languages respectively, on our synthetic dataset.
Our contributions are summarized as follows: We (1) apply a single synthetic data generation method to different language pairs, including lowresource languages such as Tamil, based on a single dataset and thereby eliminate differences that emerge from the discrepancies in data generation methodology, (2) release a new code-switching dataset, CoVoSwitch, with similar code-switching levels across 13 languages, and (3) compare trans-
lation performance in code-switching versus monolingual settings and high-resource versus lowresource languages and identify the off-target problem and hallucinations. To the best of our knowledge, this is the first work to leverage prosodic segmentation features to create a dataset containing code-switched text.

heading: 2 Synthetic Data Generation

heading: 2.1 Intonation Unit Detection
text: We use the En→X subset of the CoVoST 2 dataset, as this subset contains English recordings that we use to detect English prosodic boundaries. For nonEnglish languages, we select Arabic (ar), Catalan (ca), Welsh (cy), German (de), Estonian (et), Persian (fa), Indonesian (id), Latvian (lv), Mongolian (mn), Slovenian (sl), Swedish (sv), Tamil (ta), and Turkish (tr). We follow the classification scheme of (Costa-jussà et al., 2022) and denote Welsh, Mongolian, and Tamil as low-resource and others as high-resource. To match units of measurement for metrics such as CMI and SPF detailed later in this study, we exclude Chinese and Japanese, which are not whitespace separated. Further information on languages covered is contained in Appendix A.1.
Using the PSST model2 (Roll et al., 2023) finetuned from OpenAI’s Whisper3 (Radford et al., 2023), we both generate transcriptions and detect intonation unit (IU) boundaries for English utterances in the original Common Voice 4.0 Corpus
2https://github.com/nathan-roll1/psst 3https://huggingface.co/openai/whisper-large-v3
(Ardila et al., 2020), which serve as audio files for CoVoST 2. All English audio files were resampled at a sampling rate of 16,000 Hz to generate transcriptions with PSST. Of these, we extract sentences that contain intonation unit boundaries and exclude wrong transcriptions and outputs that contain hallucinations. Table 1 details the number of utterances used in each step, while Table 2 captures descriptive statistics on utterances used in the generated dataset.

heading: 2.2 Alignment Extraction and Intonation Unit Replacement
text: We obtain word alignments between English and non-English text from CoVoST 2 using an aligner following previous research (Rizvi et al., 2021; Winata et al., 2019; Pratapa et al., 2018), but replace fast_align (Dyer et al., 2013), a reparametrization of IBM Model 2, with a neural aligner, awesome-align4 (Dou and Neubig, 2021), because it outperforms fast_align in alignment error rate. This aligner supports all target languages covered in this work as it is a fine-tuned aligner from mBERT (Devlin et al., 2019).
We pick the number of intonation units to replace, r, from 1 to number of English intonation units - 1 for each English sentence. For each r, we randomly select a combination of r intonation unit indices, but nonconsecutive IU indices, if they exist, are prioritized over consecutive ones to represent more active code-switching. For each of the tokens in each replacement intonation unit selected, we find corresponding non-English tokens using word alignments. When replacing English tokens with non-English tokens, we preserve the original order in non-English languages. If no tokens are mapped by the aligner, empty strings are appended to the code-switched text, following previous work (Pratapa et al., 2018). For tokens that are not in the intonation units selected for replacement, English
4https://github.com/neulab/awesome-align

heading: ISO Count %L1 %L2 CMI SPF
text: tokens are appended. Once the code-switched text is created, we perform checks to ensure that the synthesized text contains at least one intonation unit from both languages. Additionally, if the resulting code-switched text is exactly equal to the source English sentence, which occurs when tokens replaced are language-independent tokens such as proper nouns present in both component languages, we do not add the code-switched text to our dataset. Figure 1 outlines an example synthesis process.

heading: 2.3 Dataset Evaluation and Analysis
text: To evaluate our synthetic dataset, we report two automatic metrics, Code Mixing Index (CMI) and Switch Point Fraction (SPF). These metrics can be computed at either the utterance or corpus level, but we report at the corpus level to facilitate parallel understanding across languages.
CMI, first proposed by (Das and Gambäck, 2014), measures the level of code-switching in a text. We follow the definition of (Mondal et al., 2022) and report CMI as follows. For a codeswitching sentence comprised of η tokens, with η1 and η2 tokens in each language and η = η1+η2, CMI is defined as 1 - max(η1,η2)η . We adhere to previous convention and multiply this number by 100. SPF was proposed by (Pratapa et al., 2018) and measures the rate at which code-switching points occur in the code-switched text. SPF is defined as ∑η−2 i=0 S(i,i+1)
η−1 where S(i, i + 1) is an indicator variable that is equal to 1 if the tokens of indices i and i+ 1 belong to different languages and else 0.
Table 3 captures information relevant to the test subset of our synthesized dataset, which is the only subset that we utilize in the experiments that follow.
The total number of sentences generated is roughly 1.5 times the number of correct transcripts used in Table 1, which is related to the average number of intonation units outlined in Table 2. CMI values range from 32.76 to 33.71, which is comparable to CMI levels of 31.00 in (Pratapa et al., 2018). SPF values range from 0.15 to 0.18, which is comparable to SPF values of 0.17 and 0.2 in (Winata et al., 2019). Because our dataset is created by replacing entire intonation units instead of words as in previous works, it contains longer same language spans and less switch points, resulting in relatively higher CMI values and lower SPF values. In our dataset, roughly half of the tokens come from each constituent language. Statistics on train and validation subsets are included in Appendix A.2.

heading: 3 Machine Translation Experimental Setup
text: Models. We use the HuggingFace pre-trained model checkpoints facebook/m2m100_418M and facebook/nllb-200-distilled-600M for the M2M-100 418M and NLLB-200 600M models. These two models were chosen for their exceptional multilingual capabilities, with M2M-100 intended for non-English centric translation and NLLB-200 designed to improve translation performance in low-resource languages. Both support all languages covered by our synthetic dataset. Translation Settings. We experiment with four translation settings for each of the English and nonEnglish language pairs. First is csw→En, in which code-switched text is translated into English. This setting was examined in previous research (Nguyen et al., 2023; Xu and Yvon, 2021), but we also experiment with csw→X to analyze any performance gaps that may arise by setting target language for translation differently. We compare these two codeswitching translation settings to two monolingual translation settings, X→En and En→X, where X is a non-English language and En is English. Baselines. Our baselines are twofold. First, we compare code-switching translations with monolingual translations and interpret deltas from monolingual baselines as the gains or losses from introducing code-switching units. We set our second baseline in consideration of our synthetic code-switched inputs. Because synthetic code-switched inputs already contain segments from reference texts, evaluation scores for these may be higher than translations of solely monolingual texts. In light of
this, we consider deltas from raw code-switched inputs the performance of systems in translating code-switched text. Evaluation Metrics. We measure the performance of translation models with the following automatic metrics: chrF++ (Popović, 2017) at the character level, spBLEU (Goyal et al., 2022) at the languageagnostic subword level tokenized through SentencePiece (Kudo and Richardson, 2018), and COMET (Rei et al., 2020) at the detokenized representation level. spBLEU and chrF++ measure similarity between reference translation and system translation, while COMET predicts human judgments of system translations based on a neural model. We use the FLORES-200 (Costa-jussà et al., 2022) tokenizer available through SacreBLEU (Post, 2018) for spBLEU and Unbabel/wmt22-comet-da (Rei et al., 2022) for COMET calculation.
We supplement chrF++, spBLEU, and COMET with copy and replacement rates to examine whether translation systems can perform implicit language identification to copy or replace tokens as appropriate. As in (Liu et al., 2021; Xu and Yvon, 2021; Song et al., 2019), we define copy rate as the rate at which the target tokens already present in code-switched input is successfully transferred over to the machine translation system output. We define replacement rate as the rate at which the system successfully converts non-target input tokens to target tokens. It follows that lower replacement rates indicate less translated outputs.
All experiments are conducted on a single NVIDIA L4 GPU.

heading: 4 Results and Discussion

heading: 4.1 Code-Switched Inputs Relative to Monolingual Translations
text: Results are shown in Table 4. Inspection of spBLEU in the to English setting reveals that 12 out of 13 synthetic code-switched inputs score higher than M2M-100 translation outputs when evaluated against reference English texts. For NLLB-200, however, only 5 code-switched inputs score higher than monolingual translations. In contrast, in the to non-English setting, raw inputs score higher than monolingual translations for 11 and 10 languages. We thus reaffirm the findings of (Nguyen et al., 2023) that code-switched inputs score higher than monolingual translations but with qualifications that exceptional monolingual translations by stronger models can outperform code-switched in-
puts and that this assertion holds more true for the to non-English setting than the to English setting.
Further, we observe that in spBLEU and chrF++ for low-resource languages such as Welsh, Mongolian, and Tamil, gaps between scores for raw codeswitched inputs and monolingual translations are larger, mainly due to worse performance of models in translating these languages. M2M-100 struggles with translation across all three languages, while NLLB-200 shows better translations. COMET scores similarly suggest that M2M-100 shows weak performance in Welsh, Mongolian, and Tamil, as they are the only languages with COMET scores under 80 in both monolingual translation settings.

heading: 4.2 Deltas Relative to Monolingual Baselines
text: Inclusion of code-switched units results in better translation than monolingual settings. This is seen in the predominantly positive deltas across spBLEU and chrF++ in Table 5. In particular, whether the languages are low-resource or high-
resource, spBLEU scores increase across all languages, models, and translation settings. We notice similar trends in chrF++ with all scores increasing for csw→X. For csw→En, some minimal decreases are observed for M2M-100 in highresource languages, while all scores increase for NLLB-200. However, improvements can be made, as deltas for COMET scores are smaller than in other metrics.
Low-resource languages gain most in csw→En and but much less in csw→X. In csw→En translation in Table 5, low-resource languages benefit the most with two-digit gains from monolingual translations, whereas high-resource languages show smaller gains. This is most prominent in M2M-100 when translating into English. Tamil, Welsh, and Mongolian show the most gains with spBLEU increases of 31.1, 27.0, and 26.9 each, while German and Swedish increase by 2.6 and 2.8. Welsh for NLLB-200 is ranked penultimately, but we regard this as trivial as spBLEU scores for
NLLB-200 have a very high average gain of 23.2 and a low standard deviation of 2.7. However, for low-resource csw→X translation, gains from monolingual are much smaller than in csw→En. In M2M-100, csw→X deltas are halved or more than halved from csw→En deltas for Welsh, Mongolian, and Tamil, while csw→X deltas become significantly larger for high-resource languages such as German, Estonian, and Latvian. In NLLB-200 csw→X translation, all low-resource languages show one digit spBLEU and chrF++ deltas. NLLB200 benefits particularly little in Welsh given the 2.7 increase in spBLEU and 1.1 increase in chrF++. This extends findings of (Goyal et al., 2022) that translating into low-resource languages is harder than translating out of them. Table 7 summarizes two languages with the most and least gains in
spBLEU for each model and setting.

heading: 4.3 Deltas Relative to Code-Switched Input Baselines
text: Models are better in code-switching translation into English than non-English. (Goyal et al., 2022) established that multilingual translation models are better at translation into English than into
non-English languages. We confirm similar results in code-switching settings. This is most evident in Table 6 with gains in performance for chrF++ and spBLEU for NLLB-200, where differences in deltas between csw→En and csw→X are double digits for the majority of the languages. High-resource languages gain further while lowresource languages lose performance gained through code-switched inputs in csw→X. Performance already gained from code-switched input is lost in low-resource languages for csw→X translation, whereas translations for high-resource languages effectively use code-switched inputs to result in even greater gains than those seen in csw→En translation. For instance, deltas of chrF++ scores in M2M-100 Catalan translation are 5.1 in csw→En and 19.0 in csw→X, compared to values in Welsh of -2.1 in csw→En and -17.2 in csw→X. Similar sized drops are seen for csw→X in Tamil with -13.7 and Mongolian with -9.2. Comparatively, NLLB-200 performs better, but the increase in csw→X in Mongolian is a mere 0.2 compared to 23.3 in Indonesian. NLLB-200 spBLEU scores yield similar conclusions, with a drop of 9.5 observed in Mongolian compared to an increase of 18.7 in Indonesian and 12.3 in Swedish. Overall, negative deltas for csw→X translation suggest that there is room for improvement for code-switching translation into non-English languages.

heading: 4.4 Analysis of Translations
text: Copy Rates. We report copy rates in Table 8. For csw→En translation, models show high copy rates ranging from 90.4 to 94.5 percent for M2M-100 and 95.5 to 98.4 percent for NLLB-200. This is
in line with findings of (Xu and Yvon, 2021) in which high copy rates were observed for csw→En translations, with code-switched text created using English, French, and Spanish. Conversely, for csw→X, models show less competent copy rates. In particular, M2M-100 exhibits copy rates of around only 50 percent for Welsh and Mongolian and below 50 percent for Tamil. NLLB-200 obtains better performance with Welsh and Tamil, but still shows weak performance for Mongolian at 55.5 percent. Copy rates for csw→X are worse than csw→En for every language and model except for M2M-100 in Indonesian and Swedish. Replacement Rates. As in copy rates, replacement rates are also generally lower for csw→X translation than csw→En translation, shown in Table 9. Here, however, models demonstrate very high performance in csw→X for languages such as Arabic, Persian, Mongolian, and Tamil, comparable to csw→En translation. In contrast, they show worse performance in csw→X with Latin scripts such as in Estonian or Turkish. We conjecture that scripts may be related to replacement rates, but leave this to be validated by future works.
Deltas from monolingual replacement rates are also reported in Table 9. Replacement rates in codeswitching translations are generally lower than those in monolingual translations. In the very occasional cases where code-switching translation replacement rates are higher, margins are very small, with the largest at 1.7 percent. Off-target Problem and Hallucination. Low replacement rates in csw→X translation suggest that
a considerable fraction of words are not being translated, despite target language being specified. Table 9 indicates that up to 33.8% of English tokens are not translated into Indonesian with M2M100 and up to 32.7% of English tokens are not translated into Indonesian with NLLB-200. Figure 2 shows examples of fully and partially translated system outputs in Catalan-English and WelshEnglish. Words in orange are code-switched tokens that remain in the system output of multilingual machine translation models. We believe this points to a case of the off-target problem seen in massively multilingual translation models (Zhang et al., 2020; Liu et al., 2023; Chen et al., 2023; Guerreiro et al., 2023), studied primarily in monolingual translation settings thus far. In our code-switching translation experiments, models ignore the specified target language and instead copy the code-switched input as the translation output.
Recent work (Tan and Monz, 2023) demonstrated that the off-target problem is a symptom rather than a cause of poor zero-shot translation in monolingual settings. To understand this in the code-switching context, we apply their methods and measure the correlation between replacement rates and spBLEU deltas relative to raw codeswitched inputs, shown in Figure 3. While there is a slight negative correlation, spBLEU deltas for replacement rates of 100% vary significantly. We therefore conclude that replacement rates are likewise not direct causes of poor code-switching translation, in accordance with prior findings.
Figure 2 also illustrates a case of hallucination. In the Welsh-English NLLB-200 translation, the words in green, Whey and crempagai, are absent in the original Welsh and English sentences. We observe, however, that the model attempted to translate or scramble the Welsh words given the similarity of Wyau and Whey and crempogau and crempagai. In addition, this demonstrates the off-target problem as models were tasked with translation into English. Hallucinations observed in csw→X translation are included in Appendix A.3.

heading: 5 Conclusion
text: In this work, we present CoVoSwitch, a codeswitching dataset created by replacing intonation units detected by PSST, a speech segmentation model fine-tuned from Whisper, on CoVoST 2, a speech-to-text translation dataset. Using CoVoSwitch, we examine the performance of two
MNMT models with millions of parameters, M2M100 418M and NLLB-200 600M, and compare code-switching translations against monolingual translations and high-resource languages against low-resource languages. We discover that the introduction of code-switching units results in higher performing translations compared to monolingual settings and that models are better at codeswitching translation into English than into nonEnglish. Meanwhile, low-resource languages gain most from monolingual baselines compared to other languages in csw→En but much less in csw→X. Systems also exhibit poor translation abilities in low-resource csw→X translation to the extent that performance already gained from codeswitched inputs is lost. Additionally, we find that models struggle to copy non-English tokens, identify the off-target problem in code-switching settings, and confirm that models hallucinate in codeswitching translation by creating words nonexistent in the original source sentences. By releasing CoVoSwitch, we aim to support the inclusion of a wider variety of languages in code-switching research.

heading: Limitations
text: We used English as the matrix language following the Matrix Language Frame Model and detected English intonation units. Future work could explore code-switching based on intonation unit replacement on languages other than English and analyze any translation performance differences from this work. Alternative methods for intonation unit replacement could also be studied for scriptio continua languages that we excluded for cross-lingual comparative analysis.

heading: Ethics Statement
text: This work does not pose ethical issues. All datasets and models used in this study are publicly available and were used under their respective Creative Commons licenses.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments and suggestions.

heading: A Appendix

heading: A.1 Languages in the Synthesized Dataset
text: We report the ISO 639-1 code, language name, family, subgrouping, script, and resource level for the 13 languages that we incorporated from CoVoST 2 in Table 12. We draw the information on language family, subgrouping, script, and resource level from (Costa-jussà et al., 2022). (Costa-jussà et al., 2022) indicates resource level with either high or low.

heading: A.2 Statistics on Train and Validation Subsets
text: We include statistics on train and validation subsets of CoVoSwitch, created from the train and validation subsets of CoVoST 2 in Tables 10 and 11.

heading: A.3 Hallucinations in csw→X Translation
text: Hallucinations, as shown in the csw→En setting in Figure 2, are also seen in csw→X. As such, we provide a few observations of the problem in Welsh-English in Figures 4 and 5. Besides the hallucination of creating words noted in Figure 2, we find repetitions of the same word. Additionally, we observe that even if two different code-switching sentences share the same source sentences, translation results can be significantly different, as seen in NLLB-200 outputs with one yielding repeated words with no meaning and the other translated but also including the repeated word Mae, highlighted in pink.
Besides repetition of words, single characters or specific combinations of characters can be repeated, as highlighted in pink in Figure 5. We note that the combination repeated here, wch, is absent in both English and Welsh source sentences and does not hold meaning relevant to the context. We find that M2M-100 not only fails to translate the English portion of the text but also completely changes its meaning when translating, from I do not like sushi to I’m not like sushi. This is also an example of the off-target problem because of the failure of the model to translate English to Welsh.

heading: A.4 Parallel Examples of Code-Switching Sentences Generated
text: All code-switched texts in CoVoSwitch are made from parallel corpora in the En→X subset of CoVoST 2, and so are created using the same set of English sentences. As a result, code-switched sentences across languages share English fragments. We include an example from the test subset in Figure 6. For some languages, we demonstrate different intonation unit replacements than others to illustrate how resulting code-switched texts diverge based on which intonation units are selected.

heading: Future Work
text: Future work could explore code-switching based on intonation unit replacement on languages other than English and analyze any translation performance differences from this work. Alternative methods for intonation unit replacement could also be studied for scriptio continua languages that we excluded for cross-lingual comparative analysis