# Code-Switching in Automatic Speech Recognition: The Issues and Future Directions
## 4 Results of Review
#### 4.2.2 Acoustic and Language Modelling  
Most papers that investigate CS in speech are for bilingual ASR [5; 6; 29; 32; 40; 41; 47; 48; 50; 51; 53; 55; 56; 58], while the rest look at multilingual ASR [4; 26; 30; 39; 52; 59]. The acoustic modelling for speech recognition is a critical process in any ASR system development. Over the years, many techniques have been employed to develop the acoustic model for monolingual, bilingual, and multilingual speech recognition systems. The development of an acoustic model for the bilingual and multilingual speech recognition systems that contain CS required the model to switch language during the recognition of speech, which can happen within a sentence. The CS phone sets in bilingual and multilingual speech recognition can be made in three distinctive ways; (1) by combining the phone sets of the two languages, (2) by mapping the phone sets of the two languages, and (3) by merging similar phone sets of the two languages [2; 42]. * Combining For both the bilingual and multilingual ASR systems, researchers prefer to combine the acoustic models of each monolingual model [4; 6; 29; 39; 40; 41; 49; 51; 53; 56; 58; 59; 62]. Combining the monolingual models for CS is common for well-resourced languages such as English, Mandarin, and German, among others. [PERSON] et al. [40] proposed the recurrent neural network language modelling toolkit for CS for bilingual ASR. Textual features such as words and part-of-speech (POS) tags were added to the input layer, while a set of all possible languages was added to the output layer. The probability for the succeeding language to be computed is based on the current word, the current features, and the history of words and features. According to [PERSON] et al. [40], recurrent neural network language models improve the perplexity and error rates compared to the traditional n-gram approaches, as the former can handle more extended contexts. On top of that, linguistic analyses of the CS help better understand the task and challenges and thus allow researchers to develop an appropriate language model [40]. [PERSON] et al. [39] developed acoustic and language models for five languages with CS for multilingual ASR. Unlike past works that only consider bilingual CS, the developed ASR system can hypothesise CS word sequences for five languages. They use the transcriptions of the CS speech data for training the language model [39]. Alternatively, [PERSON] et al. [4] trained several multilingual deep neural network (DNN) models in Frisian, English, and Dutch for developing the CS acoustic model for low-resourced languages. * Merging  
Merging of acoustic models was applied in CS involving low-resourced languages [4; 52; 5]. The merging of similar phones of the two languages for CS reduces the size and complexity of the CS. For example, ref. [52] used the union character set for each language and eliminated the duplication of output symbols in multiple languages, allowing them to reduce the computational cost during model development. When recognising CS utterances, the model can switch the language of the output sequence. The bi-directional encoder network computes the hidden representations as input acoustic features, allowing the model to predict the language identification for variable-length segments [52]. In ref. [26], a spectrogram extracted by the short-time Fourier transform (STFT) was used together with the Librosa library for generating the speech features. They first applied wave-normalisation per utterance, followed by pre-emphasis, and extracted the spectrogram with an STFT. The final set included 40 dims log mel-spectrogram features, and 1025 dims log magnitude spectrograms. On the other hand, ref. [5] applied the recurrent neural network trained with cross-lingual embedding data to maximise the use of the available textual resources. * Mapping  
Phone mapping was applied in [48], where they mapped Indonesian with Arabic phone sets.