The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges
Year: 2023

Abstract:
Code-Switching, a common phenomenon in written text and conversation, has been studied over decades by the natural language processing (NLP) research community. Initially, code-switching is intensively explored by leveraging linguistic theories and, currently, more machine-learning oriented approaches to develop models. We introduce a comprehensive systematic survey on code-switching research in natural language processing to understand the progress of the past decades and conceptualize the challenges and tasks on the codeswitching topic. Finally, we summarize the trends and findings and conclude with a discussion for future direction and open questions for further investigation.

heading: 1 Introduction
text: Code-Switching is the linguistic phenomenon
CSW since the early 1900s, we also investigate to what extent theoretical frameworks from these fields have influenced NLP approaches (see §5), and how the choice of methods has evolved over time (see §5.4). Finally, we discuss the most pressing research challenges and identify a path forward to continue advancing this exciting line of work (see §6).
The area of NLP for CSW data is thriving, covering an increasing number of language combinations and tasks, and it is clearly advancing from a niche field to a common research topic, thus making our comprehensive survey timely. We expect the survey to provide valuable information to researchers new to the field and motivate more research from researchers already engaging in NLP for CSW data.

heading: 2 Exploring Open Proceedings
text: To develop a holistic understanding of the trends and advances in CSW NLP research, we collect
research papers on CSW from the ACL Anthology and ISCA proceedings. We focus on these two sources because they encompass the top venues for publishing in speech and language processing in our field. In addition, we also look into personal repositories from researchers in the community that contains a curated list of CSW-related papers. We discuss below the search process for each venue.
ACL Anthology We crawled the entire the ACL Anthology repository up to October 2022.1
We then filtered papers by using the following keywords related to CSW: “codeswitch", “code switch", “code-switching", “code-switched", “code-switch", “code-mix", “code-mixed", “codemixing", “code mix" “mixed-language", “mixedlingua", “mixed language", “mixed lingua", and “mix language".
ISCA Proceedings We manually reviewed publicly available proceedings on the ISCA website2
and searched for papers related to CSW using the same set of keywords as above.
Web Resources To extend the coverage of paper sources, we also gathered data from existing repositories.3,4 We can find multiple linguistics papers studying about CSW.

heading: 2.1 Annotation Process
text: We have three human annotators to annotate all collected papers based on multiple categories shown in Table 1. All papers are coded by a least one
1https://github.com/acl-org/ acl-anthology
2https://www.isca-speech.org 3https://github.com/gentaiscool/
code-switching-papers 4https://genius1237.github.io/emnlp19_ tut/
annotator. To extract the specific information we are looking for from the paper, the annotator needs to read through the paper, as most of the information is not contained in the abstract. The full list of the annotations we collected is available in the Appendix (see §A).
To facilitate our analysis, we annotated the following aspects:
• Languages: CSW is not restricted to pairs of languages; thus, we divide papers by the number of languages that are covered into bilingual, trilingual, and 4+ (if there are at least four languages). For a more fine-grained classification of languages, we categorize them by geographical location (see Figure 2).
• Venues: There are multiple venues for CSWrelated publications. We considered the following type of venues: conference, workshop, symposium and book. As we will discuss later, the publication venue is a reasonable data point of the wider spread of CSW research in recent years.
• Papers: We classify the paper types based on their contribution and nature. We predict that we will have a high distribution of dataset/resource papers, as lack of resources has been a major bottleneck in the past.
• Datasets: If the paper uses a dataset for the research, we will identify the source and modality (i.e., written text or speech) of the dataset.
• Methods: We identify the type of methods presented in work.
• Tasks: We identify the downstream NLP tasks (including the speech processing-related tasks) presented in work.

heading: 3 Language Diversity
text: Here, we show the languages covered in the CSW resources. While focusing on the CSW phenomenon increases diversity of NLP technology, as we will see in this section, future efforts are needed to provide significant coverage of the most common CSW language combinations worldwide.

heading: 3.1 Variety of Mixed Languages
text: Figure 3 shows the distribution of languages represented in the NLP for CSW literature. Most of the papers use datasets with two language pairs. However, we did find a few papers that address CSW scenarios with more than two languages. We
Languages # Publications non-ST ST Total
Language Identification 46 17 63 Sentiment Analysis 31 30 61 NER 17 14 31 POS Tagging 29 1 30 Abusive/Offensive Lang. Detection 9 16 25 ASR 20 0 22 Language Modeling 19 1 20 Machine Translation 8 5 13
# Publications *CL ISCA Total
Public Dataset 38 4 42 Private Dataset 54 18 72

heading: 4 Tasks and Datasets

heading: 3.2 Language-Dialect Code-Switching
text: Source *CL ISCA Total
Social Media 183 3 186 Speech (Recording) 29 102 141 Transcription 23 4 27 News 19 5 24 Dialogue 16 2 18 Books 7 1 8 Government Document 6 0 6 Treebank 5 0 5
Type *CL ISCA Total
Empirical 205 100 305 Shared Task 82 1 83 Corpus (Closed) 54 18 62 Corpus (Open) 38 4 42 Analysis 34 8 42 Demo 7 2 9 Theoretical/Linguistic 7 0 7 Position/Opinion/Survey 3 0 3 Metric 2 1 3

heading: 5.1 Linguistic-Driven Approaches

heading: 5 From Linguistics to NLP
text: CSW data with synthetic CSW data can effectively improve the perplexity. They also treat parallel sentences as a linear structure and only allow switching on non-crossing alignments.
Matrix-Embedded Language Framework (MLF) Myers-Scotton (1997) proposed that in bilingual CSW, there exists an asymmetrical relationship between the dominant matrix language and the subordinate embedded language. Matrix language provides the frame of the sentence by governing all or most of the most of the grammatical morphemes as well as word order, whereas syntactic elements that bear no or only limited grammatical function can be provided by the embedded language (Johanson, 1999; Myers-Scotton, 2005). Lee et al. (2019a) use augmented parallel data by utilizing MLF to supplement the real code-switch data. Gupta et al. (2020) use MLF to automatically generate the code-mixed text from English to multiple languages without any parallel data.
Functional Head Constraint Belazi et al. (1994) posit that it is impossible to switch languages between a functional head and its complement because of the strong relationship between the two constituents. Li and Fung (2014) use the constraint of the LM by first expanding the search network with a translation model and then using parsing to restrict paths to those permissible under the constraint.

heading: 5.2 Learning from Data Distribution
text: Linguistic constraint theories have been used for decades to generate synthetic CSW sentences to address the lack of data issue. However, the approach requires external word alignments or constituency parsers that create erroneous results instead of applying the linguistic constraints to generate new synthetic CSW data, building a pointergenerator model to learn the real distribution of code-switched data (Winata et al., 2019c). Chang et al. (2019) propose to generate CSW sentences from monolingual sentences using Generative Adversarial Network (GAN) (Goodfellow et al., 2020) and the generator learns to predict CSW points without any linguistic knowledge.

heading: 5.3 The Era of Statistical Methods
text: The research on CSW is also influenced by the progress and development of machine learning.
According to Figure 5, starting in 2006, statistical methods have been adapted to CSW research, while before that year, the approaches were mainly rule-based. There are common statistical methods for text classification used in the literature, such as Naive Bayes (Solorio and Liu, 2008a) and Support Vector Machine (SVM) (Solorio and Liu, 2008b). Conditional Random Field (CRF) (Sutton et al., 2012) is also widely seen in the literature for sequence labeling, such as Part-of-Speech (POS) tagging (Vyas et al., 2014), Named Entity Recognition (NER), and word-level language identification (Lin et al., 2014; Chittaranjan et al., 2014; Jain and Bhat, 2014). HMM-based models have been used in speech-related tasks, such as speech recognition (Weiner et al., 2012a; Li and Fung, 2013) and text synthesis (Qian et al., 2008; Shuang et al., 2010; He et al., 2012).

heading: 5.4 Utilizing Neural Networks
text: Following general NLP trends, we see the adoption of neural methods and pre-trained models growing in popularity over time. In contrast, the statistical and rule-based approaches are diminishing. Compared to ISCA, we see more adaptation of the pre-training model. This is because ACL work is more text-based focused, where pre-trained LMs
are more widely available.
Neural-Based Models Figure 5 shows that the trend of using neural-based models started in 2013, and the usage of rule/linguistic constraint and statistical methods diminished gradually through time, but they are still used even with a low percentage. RNN and LSTM architectures are commonly used in sequence modeling, such as language modeling (Adel et al., 2013; Vu and Schultz, 2014; Adel et al., 2014c; Winata et al., 2018a; Garg et al., 2018a; Winata et al., 2019c) and CSW identification (Samih et al., 2016a). DNN-based and hybrid HMM-DNN models are used in speech recognition models (Yilmaz et al., 2018; Yılmaz et al., 2018).
Pre-trained Embeddings Pre-trained embeddings are used to complement neural-based approaches by initializing the embedding layer. Common pre-trained embeddings used in the literature are monolingual subword-based embeddings, FastText (Joulin et al., 2016), and aligned-embeddings MUSE (Conneau et al., 2017). A standard method to utilize monolingual embeddings is to concatenate or sum two or more embeddings from different languages (Trivedi et al., 2018). A more recent approach is to apply an attention mechanism to merge embeddings and form metaembeddings (Winata et al., 2019a,b). Characterbased embeddings have also been explored in the literature to address the out-of-vocabulary issues on word-embeddings (Winata et al., 2018b; Attia et al., 2018; Aguilar et al., 2021). Another approach is to train bilingual embeddings using real and synthetic CSW data (Pratapa et al., 2018b). In the speech domain, Lovenia et al. (2022) utilize wav2vec 2.0 (Baevski et al., 2020) as a starting model before fine-tuning.
Language Models Many pre-trained model approaches utilize multilingual LMs, such as mBERT or XLM-R to deal with CSW data (Khanuja et al., 2020b; Aguilar and Solorio, 2020; Pant and Dadu, 2020; Patwa et al., 2020; Winata et al., 2021a). These models are often fine-tuned with the downstream task or with CSW text to better adapt to the languages. Some downstream fine-tuning approaches use synthetic CSW data due to a lack of available datasets. Aguilar et al. (2021) propose a character-based subword module (char2subword) of the mBERT that learns the subword embedding that is suitable for modeling the noisy CSW text. Winata et al. (2021a) compare the performance of
the multilingual LM versus the language-specific LM for CSW context. While XLM-R provides the best result, it is also computationally heavy. There needed to be more exploration of larger models.
We see that pre-trained LMs provide better empirical results on current benchmark tasks and enables an end-to-end approach. Therefore, one can theoretically work on CSW tasks without any linguistic understanding of the language, assuming the dataset for model finetuning is available. However, the downside is that there is little understanding of how and when the LMs would fail, thus we encourage more interpretability work on these LMs in CSW setting.

heading: 6 Recent Challenges and Future Direction

heading: 6.1 More Diverse Exploration on Code-Switching Styles and Languages
text: A handful of languages, such as Spanish-English, Hindi-English, or Chinese-English, dominate research and resource CSW. However, there are still many countries and cultures rich in the use of CSW, which is still under-represented in NLP research (Joshi et al., 2020; Aji et al., 2022; Yong et al., 2023), especially on different CSW variations. CSW style can vary in different regions of the world, and it would be interesting to gather more datasets on unexplored and unknown styles, which can be useful for further research and investigation on linguistics and NLP. Therefore, one future direction is to broaden the language scope of CSW research.

heading: 6.2 Datasets: Access and Sources
text: According to our findings, there are more than 60% of the datasets are private (see Table 4), and they are not released to the public. This eventually hampers the progress of CSW research, particularly in the results’ reproducibility, credibility, and transparency. Moreover, many studies in the literature do not release the code to reproduce their work. Therefore, we encourage researchers who build a new corpus to release the datasets publicly. In addition, the fact that some researchers provide urls to download the data is also problematic due to the data attrition issue we raised earlier. Data attrition is bad for reproducibility, but it is also a waste of annotation efforts. Perhaps we should work on identifying alternative means to collect written CSW data in an ecologically valid manner.

heading: 6.3 Model Scaling
text: To the best of our knowledge, little work has been done on investigating how well the scaling law holds for code-mixed datasets. Winata et al. (2021a) demonstrate that the XLM-R-large model outperforms smaller pre-trained models on the NER and POS tasks in LinCE benchmark (Aguilar et al., 2020); however, the largest model in the study, which is the XLM-R-large model, only has 355 million parameters. Furthermore, they find that smaller models that combine word, subword, and character embeddings achieve comparable performance as mBERT while being faster in inference. Given the recent release of billion-sized large pre-trained multilingual models such as XGLM and BLOOM (Scao et al., 2022), we urge future research to study the scaling law and performancecompute trade-off in code-mixing tasks.

heading: 6.4 Zero-Shot and Few-Shot Exploration
text: The majority of pre-trained model approaches fine-tune their models to the downstream task. On the other hand, CSW data is considerably limited. With the rise of multilingual LMs, especially those that have been fine-tuned with prompt/instruction (Muennighoff et al., 2022; Ouyang et al., 2022; Winata et al., 2022), one direction is to see whether these LMs can handle CSW input in a zero-shot fashion. This work might also tie in with model scaling since larger models have shown better capability at zero-shot and fewshot settings (Winata et al., 2021b; Srivastava et al., 2022).

heading: 6.5 Robustness Evaluation
text: Since CSW is a widely common linguistic phenomenon, we argue that cross-lingual NLP benchmarks, such as XGLUE (Liang et al., 2020) and XTREME-R (Ruder et al., 2021), should incorporate linguistic CSW evaluation (Aguilar et al., 2020; Khanuja et al., 2020b). The reasons are that CSW is a cognitive ability that multilingual human speakers can perform with ease (Beatty-Martínez et al., 2020). CSW evaluation examines the robustness of multilingual LMs in learning cross-lingual alignment of representations (Conneau et al., 2020; Libovickỳ et al., 2020; Pires et al., 2019; Adilazuarda et al., 2022). On the other hand, catastrophic forgetting is observed in pre-trained models (Shah et al., 2020), and human speakers (Hirvonen and Lauttamus, 2000; Du Bois, 2009, known as lan-
guage attrition) in a CSW environment. We argue that finetuning LMs on code-mixed data is a form of continual learning to produce a more generalized multilingual LM. Thus, we encourage CSW research to report the performance of finetuned models on both CSW and monolingual texts.

heading: 6.6 Task Diversity
text: We encourage creating reasoning-based tasks for CSW texts for two reasons. First, code-mixed datasets for tasks such as NLI, coreference resolution, and question-answering are much fewer in comparison to tasks such as sentiment analysis, parts-of-speech tagging, and named-entity recognition. Second, comprehension tasks with the CSW text present more processing costs for human readers (Bosma and Pablos, 2020).

heading: 6.7 Conversational Agents
text: There has been a recent focus on developing conversational agents with LMs such as ChatGPT,5 Whisper (Radford et al., 2022), SLAM (Bapna et al., 2021), mSLAM (Bapna et al., 2022). We recommend incorporating the capability of synthesizing code-mixed data in human-machine dialogue, as CS is a prevalent communication style among multilingual speakers (Ahn et al., 2020), and humans prefer chatbots with such capability (Bawa et al., 2020).

heading: 6.8 Automatic Evaluation for Generation
text: With the rise of pre-trained models, generative tasks gained more popularity. However, when generating CSW data, most work used human evaluation for measuring quality of the generated data. Alternative automatic methods for CS text based on word-frequency and temporal distribution are commonly used (Guzmán et al., 2017; Mave et al., 2018), but we believe there is still much room for improvement in this respect. One possible future direction is to align the evaluation metrics to human judgement of quality (Hamed et al., 2022) where we can assess separately the “faithfulness” of the resulting CSW data from other desired properties of language generation. Other nuances here are related to the intricacy of CSW patterns, where ideally the model would mimic the CSW style of the intended users.
5https://openai.com/blog/chatgpt/

heading: 7 Conclusion
text: We present a comprehensive systematic survey on code-switching research in natural language processing to explore the progress of the past decades and understand the existing challenges and tasks in the literature. We summarize the trends and findings and conclude with a discussion for future direction and open questions for further investigation. We hope this survey can encourage and lead NLP researchers in a better direction on code-switching research.
Limitations
The numbers in this survey are limited to papers published in the ACL Anthology and ISCA Proceedings. However, we also included papers as related work from other resources if they are publicly available and accessible. In addition, the category in the survey does not include the code-switching type (i.e., intra-sentential, inter-sentential, etc.) since some papers do not provide such information.
Ethics Statement
We use publicly available data in our survey with permissive licenses. No potential ethical issues in this work.

heading: Acknowledgements
text: Thanks to Igor Malioutov for the insightful discussion on the paper.

heading: A Annotation Catalog
text: We release the annotation of all papers we use in the survey.
A.1 *CL Anthology Bilingual Table 7 shows the annotation for papers with African-English. Table 8 shows the annotation for papers with East Asian-English languages. Table 9 shows the annotation for papers with European-English languages. Table 10 shows the annotation for papers with Middle EasternEnglish languages. Table 11 and Table 12 show the annotation for papers with South Asian-English languages. Table 13 shows the annotation for papers with South East Asian-English languages. Table 14 shows the annotation for papers with a combination of a language with a dialect. Table 15 shows the annotation for papers with languages in the same family. Table 16 shows the annotation for papers with languages in different families.
Trilingual Table 17 shows the annotation for papers with three languages.
4+ Table 18 shows the annotation for papers with four or more languages.
A.2 ISCA Proceeding Bilingual Table 19 shows the annotation for papers with African-English. Table 20 shows the annotation for papers with East Asian-English languages. Table 21 shows the annotation for papers with European-English languages. Table 22 shows the annotation for papers with Middle EasternEnglish languages. Table 23 shows the annotation for papers with South Asian-English languages. Table 24 shows the annotation for papers with South East Asian-English languages. Table 25 shows
the annotation for papers with a combination of a language with a dialect. Table 26 shows the annotation for papers with languages in the same family. Table 27 shows the annotation for papers with languages in different families.
Trilingual Table 28 shows the annotation for papers with three languages.
4+ Table 29 shows the annotation for papers with four or more languages.
Paper Proceeding SEA Mandarin-English Bangla-Chinese-Dutch- Early New High German, Telugu, MSA, Berber, French, Others (4+) English, Swiss German Algerian, MSA, Mandarin-HakkaEnglish-Farsi-German- Latin, French, Greek, Modern Standard Telugu, local Algerian Arabic Latin local Arabic varieties, Taiwanese-English Hindi-Korean-Russian- Italian, Hebrew English, Hindi, Urdu, Berber, French, and English
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section on page 9
A2. Did you discuss any potential risks of your work? Not applicable. This is a survey paper. There is no potential negative risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 on page 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work