# Code-Switching in Automatic Speech Recognition: The Issues and Future Directions
## 5 Discussion  
Bilingual and multilingual speech recognition with CS differs from monolingual speech recognition as it has its issues and challenges. Bilingual and multilingual recognition without CS is where the speech recognition system is not expected to switch from one language to another during the recognition process. These bilingual and multilingual systems usually use the existing databases for well-resourced languages [27, 28, 35, 37, 38, 44, 45, 57] and are self-developed for under-resourced languages [31, 32, 40]. For bilingual and multilingual speech recognition with CS, 24 out of 42 (representing 57%) of experimental papers consider CS in bilingual and multilingual speech recognition development. This finding shows that CS is integral in developing bilingual and multilingual speech recognition systems. The existing research focuses mainly on single-language pairs of CS, which means bilingual speech recognition. CS research aims to allow speech recognition to recognise a change in the language within a sentence uttered by multilingual users. However, this does not prevent researchers from investigating alternatives to modelling the acoustics for the CS of multiple language pairs. The issue of data sparsity was not new, as researchers faced the same issues in the early days of ASR research. However, today, monolingual ASR research has impressive resources in terms of databases, speakers, and recording size, enabling the development of more accurate acoustic models. Despite having many monolingual databases that support the development of bilingual and multilingual speech recognition systems, the uniqueness of the languages makes it impossible to have a common database for developing bilingual and multilingual speech recognition systems in a CS environment. For CS, a relatively new domain in ASR research, the issue of data sparsity affects the ability of the system to recognise CS. The data sparsity includes both the availability of CS databases and the scarcity of CS occurrences in the speech corpus due to the various combinations of languages in CS. Researchers need to develop their databases in these early days, but our review shows that many now prefer to use the existing available databases instead of developing their own ones (refer to Table 2). It may indicate that developing CS databases is resource-intensive and time-consuming, where researchers prefer to use the existing CS databases for their research. We found that the variability in the occurrence of CS is one of the reasons why CS databases are limited in terms of coverage of speech. Most CS is for bilingual ASR systems, which can have a finite mix of words from the two languages. The databases developed by researchers may not include all possible mixes, leading to sub-optimal results. The databases are also limited in terms of size and variation in speech, resulting in an imbalance of resources and tasks for developing an effective recognition model. The uneven proportion of languages in the CS database can lead to task-quantity imbalance, particularly for under-resourced languages. Another issue in CS is the need for manual annotation of the CS database, causing the development of the CS database to be expensive and time-consuming. A possible solution to manual annotation is to use the E2E method that directly converts input speech feature sequences to output label sequences without any explicit and intermediate representation of phonetic/linguistic constructs such as phonemes or words. Some researchers developed the CS database using recorded speech from sources such as broadcasts and podcasts. This form of database offers more natural CS as compared to those recorded in the studio. The main issue will be the resources needed to annotate these speeches. However, a method such as E2E allows the development of acoustic models without the need for manual annotation. Enriching the CS database from existing sources and finding an effective way to annotate speech can be one of the future directions in CS research.