# Code-Switching in Automatic Speech Recognition: The Issues and Future Directions
## 2 Research Background
### CS in Bilingual and Multilingual Speech Recognition Systems  
This section examines CS in spoken language and its role in bilingual and multilingual speech recognition systems. Many current ASR systems recognise only one language (monolingual). However, a practical ASR system must be able to handle CS, which is not an easy task as the system needs to deal with multilingual input with unpredictable switching positions [26]. Individuals that can speak more than one language (e.g., bilingual or multilingual) CS or mix their languages when communicating with others [3; 5; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39]. CS is common in bilingual and multilingual communities (different cultures and language backgrounds) [40]. CS also occurs in minority languages influenced by the majority or majority languages influenced by lingua francas, such as English and French [4]. In CS, the base language refers to the language to which the syntax of a CS sentence belongs, and the foreign words are referred to as an embedded language [41]. Code-switching can occur between two languages that differ in terms of linguistic, pronunciation and speech features (for example Mandarin and English), or have high similarity in terms of linguistic, pronunciation and speech features (for example Frisian and English). CS can be classified into two primary categories: inter-sentential and intrasentential [26; 42; 42]. The classification is important as it shows where the CS will occur in a sentence as well as the speech unit involved during the CS [6]. When CS occurs between sentences, it is referred to as inter-sentential CS, where speakers use words, phrases or sentences from one language (the embedded language) with words or sentences in their primary or base language [2]. This type of CS is common among fluent bilingual speakers. For example, \"If you are late for the job interview, see almmazsm/If you are late for the job interview, you will not be hired\" (CS of English and Turkish languages), and \"If I am late to the appointment, ignore sahaja/If I am late to the appointment, just ignore\" (CS of Malay and English languages). Intra-sentential CS is where the shift occurs in the middle of a sentence. For example, \"Wo zhidao you cong Wo de peng you/I know you from my friend\" (CS of Mandarin and English), \"No cong dang celebrate cai sinh nhat/He's also celebrating his birthday\" (CS of Vietnamese and English), \"I tak nak you campur tangan dalam life I/I don't want you to interfere in my life\" (CS of Malay and English languages). For intra-sentential CS, the units, and the locations of the switches may vary widely from single-word switches to whole phrases (beyond the length of standard loanword units). As such, a vital issue to be solved is that the speech recognition system needs to consider many context-dependent phone combinations. However, the data sparseness problem for phone modelling hinders this issue from being solved. The accuracy of ASR systems is adversely affected when dealing with CS speech. The reduction in accuracy is caused by unexpected pronunciation when languages are mixed. The embedded phonemes usually have more significant variation than the base language phonemes; either resembling the embedded language pronunciation or realised as a set of phonemic equivalents from the base language [2]. Early works on CS speech recognition employ the hybrid framework with three sub-models: a pronunciation model, an acoustic model and a language model [41; 42]. The pronunciation model takes the pronunciation variations of words in a dictionary. On the other hand, the acoustic model is a statistical model of acoustic features for sub-word units (for example, phonemes). The language model enables the ASR system to reduce the search space by using the conditional probabilities of subsequent words with the observed word sequences. Currently, these sub-models are trained and optimised separately, leading to sub-optimal results.