Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation
Authors: A. Seza Doğruöz
Sunayana Sitaram
Zheng Xin Yong
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingualism
Code-switching
representativeness
data preparation
annotation
transcription

Abstract: Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions. However, there is still not much progress in building successful CSW systems, despite the recent advances in Massive Multilingual Language Models (MMLMs). We investigate the reasons behind this setback through a critical study about the existing CSW data sets (68) across language pairs  in terms of the collection and preparation (e.g. transcription and annotation) stages. This in-depth analysis reveals that \textbf{a)} most CSW data involves English ignoring other language pairs/tuples \textbf{b)} there are flaws in terms of representativeness in data collection and preparation stages due to ignoring the location based, socio-demographic and register variation in CSW. In addition, lack of clarity on the data selection and filtering stages shadow the representativeness of CSW data sets. We conclude by providing a short check-list to improve the representativeness for forthcoming studies involving CSW data collection and preparation.

Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation
A. Seza Dogruöz
Sunayana Sitaram Microsoft Research India, Bangalore, India sunayana. si www.ttaaimiiosooooteiio com
Universiteit Gent, Gent, Belgium as. dog.ooooozeeegt.tco be
Zheng-Xin Yong
Brown University
contact, yong@broo.. edu
Abstract
in building mixed language systems which can process and produce CSW speech and text seam- lessly across different language pairs (c.g., Spanish- English and Hindi-EEglhin as exceptions). Based on an in-depth analysis of CSW data sets (68), we argue that the lack of representative CSW data col- lection and preparation procedures could lead to this drawback.
Multilingualism is widespread around the world and code-switching (CSW) is common practice among different language pairs/tuples across locations and regions. However, there is still not much progress in building suc- cessful CSW systems, despite the recent ad- vances in Massive Multilingual Language Mod- els (MMLMs). We investigate the reasons be- hind this setback through a critical study about the existing CSW data sets (68) across language pairs in terms of the collection and prepara- tion (e.g. transcription and annotation) stages. This in -e-tepth analysis reveals that a) most CSW data involves English ignoring other language pairs/tuples b) there flaws in terms of repre- sentativeness in data collection and preparation stages due to ignoring the location based, socio- demographic and register variation in CSW. In addition, lack of clarity on the data selection and filtering stages shadow the representative- ness of CSW data sets. We conclude by pro- viding a short cieck-list to improve the repre- sentativeness for forthcoming studies involving CSW data collection and preparation.
Although they claim to be multilingual and ca- pable of handling diverse sets of languages, gen- erative language models (e.g, GPT-3.5 (Ouyang et al., 2022) and BLOOM (Seao al., 2022)) per- form worse on NLP tasks concerning low low-resouree languages (Lai et al., 2023). In addition, Yong et al. (2023) show that open-source multilingual language models fail to generate CSW for South- east Asian language pairs (e.g. English-Tamil and English-Tagales Tagalog). Similarly, Zhang al. (2023) veal the performance gap between small fine-tuned models and LLMs with perrossot/ffww-hhh prompt- ing on machine translation, sentiment analysis, and language identification tasks involving texts with CSW (Spanish-English, Malayalam-English Tamil-English, Hindi-Englishh and Modern Stan- dard Arabic-Egyptian Arabic).
1 Introduction
Millions of bilingual/mulliiinuaa speakers around the world speak more than one language/dialect in their daily lives and/or mix them which (known as code-switching (CSW)). Some of these languages are also considered as low-resource (Dogruöz and Sitaram, 2022b; Aji et al., 2022). Since Solorio and Liu (2008), there is a wide range of research involving multilingual and CSW data across dif- ferent domains of computational linguistics (e.g.. Sitaram et al. (2020), Winata et al. (2022)). Further- more, research in multilingualism and CSW has of the "Next Big Ideas" at 60th Annual Meeting of the Association for Com- putational Linguistics (ACL'22).
Performance of MLLMs CSW data is still much poorer in comparison to their performances monolingual data (Khanuja al., 2020b). In addition, multilingual BERT (Devlin et al., 2019) is trained mainly on Wikipedia articles, and per- forms much worse on standard CSW benchmarks (Khanuja et al., 2020b) than XLM-R (Conneau et al., 2020). Existing CSW evaluation benchmarks may also fail to represent real-life CSW accurately. For example, ASR models that were trained to per- form well on CSW speech data tend to perform poorly on monolingual speech data in the same lan- guages and vice-versa (Shah et al., 2020). CSW benchmarks for ASR typically only contain CSW speech but not the monolingual speech which is also part of the real-life communication. ad-
been presented
Despite these encouraging prospects and avail- ability of MMLMs, there is still not much progress

---

dition, CSW evaluation data sets created through social media data (c.g. Twitter)' may also be lim- ited due to the assumptions made during the data collection and preparation stages.
before deploying the CSW dataset for training lan- guage models. In the subsections below, we ex- pand on the dimensions of variation in relation to the representativeness for multilingual and CSW data in terms of data collection and preparation procedures.
As result, language models that are overly opti- mized for some benchmarks and leaderboards have impressive results in terms of system performance. but they not very useful for the multilingual speakers/users since they do not represent CSW as it takes place in real-life communication.
2.1 Location Based Variation
Within computational approaches to CSW, there is tendency to group different varieties of the same language pairs together. However, this approach ignores the linguistic variation across locations and regions. To support this argument, provide empirical evidence using the ASCEND (Lovenia et al., 2021) and SEAME speech data sets (Lyu et al., 2010). Both of these data sets are grouped under Mandarin-Englhe
Although we analyze CSW data sets in depth. we do not aim for a literature survey describing all the tasks, experiments and their results about CSW across language pairs. Considering that labelled data is still necessary for fine-tuning MILMs, we only focus on the data collection and preparation stages to assess the issues about representativeness before the modeling stage. This assessment is not only relevant for ethical and scientific purposes but it is also a necessity for product related issues in industrial and/or social good applications which target multilingual speakers/users and their com- munities.
CSW in a recent survey (Winata et al., 2022). However, the data sets were collected in different locations (Hong Kong and Singapore) and there is variation between these two locations in terms of the historical backgrounds and language choices about multilingualism and CSW (Ng and Cavallaro, 2019).
2 Defining Representatiiveess for CSW Data
To evaluate the variation reflected on the CSW patterns between ASCEND and SEAME data sets, we trained Automatic Speech Recognition (ASR) models on both ASCEND and SEAME (sce Ap- pendix for the details of the experimental setup) data sets. As shown in Table I, we observe a sub- stantial performance gap between 25% and 35% in Match Error Rate (MER) and Character Error Rate (CER) when the models were trained and evaluated on different data sets. Even if both data sets claim to cover the same language pair (i.e., Mandarin- English), our results indicate that the CSW data collected from one region (e.g., Hong Kong) does not represent the CSW data collected from another region (e.g., Singapore) and ignoring this variation leads to system failures.
Language technologies depend on large data sets of language (i.c., corpora). Biber (1993) defines rep- resentativeness in corpora as "the extent to which a sample includes the full range of variability in a population" and it is a core requirement to be able to make generalizations about a language. Borovicka et al. (2012) define a representative data set as a special subset of an original set which is smaller in size but captures most of the information from the original set.
As illustrated by Dogruöz al. (2021), CSW patterns vary even within the same language pairs depending on various factors (e.g.. location, con- text, socio-demographic factors of speakers/users. historical factors). If this is the case, collecting ran- dom CSW data sets without taking this variation into account will lead to unrepresentative data sets without external validity (i.e., the collected data will not represent the CSW in real-life). As re- sult, CSW systems trained on unrepresentative data sets will fail to meet the needs and preferences of the target multilingual speakers/users. Therefore, we posit that researchers should perform quality measures on the representativeness of the datasets
To explore the reasons behind these results, we analyze example (1) taken from the SEAME cor- pus (Singapore) indicating CSW between Hokkien (another dialect of Chinese), Mandarin and En- glish. This example sounds different for bilingual (Mandarin English) speakers from Hong Kong since it also includes words from Hokkien (c.g., (e.g., "lah" "lor" discourse markers) which are commonly used in Mandarin-Englin informal conversations in Singapore (and Malaysia) but not in Hong Kong. The variation illustrated in this ex- ample also serves as an evidence to indicate that
'https://twitten"

---

ASCEND (Train)
SEAME (Train)
Test Datasets
Pretraining Languages
MER
1 CER
J MER
CER
Chinese (Mandarin)
26.40
22.89
53,40 (+29.0)
49.26 (+26.37)
ASCEND
English
30.33
24.17
61.23 (+30.9)
52.85 (+28.68)
Chinese (Mandarin)
65.77 (+33.51)
53.19 (430.52)
32.26
22.67
SEAME
English
64.39 (+32.65)
54.66 (+32.30)
31.74
22.36
Table I: ASR performance trained and evaluated ASCEND (Lovenia al., 2021) and SEAME (Lyu et al., 2010)
from Hong Kong and Singapore respectively. We indicate the (performance gap) in enror rate between models that are wained-and-evaluee on the same datasets (bold text) and on different datasets.
sort of hamji
lah after that sort of embarrassed dm after that (It is) kinda awkward. After all,'
clude CSW across different language pairs/tuples. Moreover, not all multilingual users (e.g., children, elderly, vulnerable minority groups) have a pres- ence on the Internet especially in low-resource and multilingual contexts (c.g., Nguyen et al. (2016), Doğruöz and Sitaram (2022a)). If these users are not present online, their language use will also not be represented in the data sets that are collected from online resources. Our claim that the lack of representativennss of CSW in internet data is strongly supported by recent findings that gener- ative MMLMs pretrained on internet data fail process and generate CSW texts in a zero-shot few-shot settings (Zhang et al., 2023; Yong et al., 2023) as well.
wō mén you shi laojiao we also are old-timers
'we are also old-timers.'
Example I: Mandarii-EEnllish-oooiii CSW from the SEAME corpus (Lyu et al., 2010). Hokkien text in- dicated with bold and Mandarin text is italicized. "dm" stands for the discourse marker (Comrie et al., 2008)
CSW data collected in one location does not repre- sent the CSW in another location even if they are grouped under the same language pair.
In addition to the location based variation for CSW in terms of countries/regions, Pratapa and Choudhury (2017a) explain how the amount of CSW varies among multilingual speakers based in urban vs. rural settings even for the same language pairs in India (e.g., Hindi-Engie Findi-Engllssh. In that sense, Hindi-English CSW data collected in rural setting may not represent the Eindi-English CSW spoken in an urban setting. Hence, ignoring this variation and overgeneralizing CSW patterns in one location other locations may lead to system failures as illustrated Table (1).
2.3 Register Variation
CSW is often associated with informal contexts in real-life communication (especially in multilin- gual immigrant communities (Çetinoglu and Çöl- tekin, 2022; Dogruöz et al., 2021)). Considering that social media language is closer to the spo- ken language in real-life communication (Herring, 1996), it is possible to encounter more examples of CSW in social media rather than written media (e.g., Wikipedia). However, there are also formal registers that include CSW patterns in multilingual communities with colonial backgrounds. For exam- ple, David (2003) illustrates language mixing be- tween English and Malay in Malaysian courtrooms as an example of CSW in formal registers. Simi- larly, Gupta et al. (2016a) indicate Hindi-Englise CSW on an online governmental platform in India. In that sense, focusing only informal registers (e.g., conversational or social media data) for CSW data collection does not capture the whole picture about multilingual language use and raises flags for representativeness for certain language pairs (c.g.. Eeaaysiin-Enngliie Hindi- Aandi--Enlish and contexts
Overgeneralizations about Internet Data Another issue about representativeness concerns the limitation of MMLMs about the coverage of languages available on the Internet.
2.2
First of all, most data on the Internet is still in En- glish (57.2% of all the webpages (Web) and 66% of the 250 Youtube channels (Yang, 2019)) with considerably fewer resources for other languages (cf. Navigli et al. (2023)) and there is no informa- tion about to what extent Internet based data in-

---

(e.g,, ex-colonial regions where English dominated the official communication).
ample, Hindi and English are widely spoken in India and they act as lingua franca. However, many speakers use these languages only certain com- munication contexts (e.g., education, work) and use other languages/dialects in their daily lives (e.g., communication with family and friends). There- fore. focusing only on Hindi-English CSW for these speakers does not fully represent their mul- tilingual abilities and CSW across different lan- guages in their daily communication.
2.4
Socoo-Demographic Variation
2.4.1
Participants
Research on multilingual and CSW communication relies on participants who act as speakers and/or users and provide data. As illustrated by Dogruöz et al. (2021) there is variation in CSW practices across multilingual speakers/users with different socio-demographic profiles (e.g., age, gender, lan- guage proficiency). In this section, we elaborate on different types of socio-demographic variation and their relation with CSW.
Secondly, not all multilingual speakers/users have similar levels of language proficiencies in the languages they claim to speak. For exam- ple, Koban (2013), Quirk (2021) and Smolak et al. (2020) observe systematic influence of language proficiency the CSW patterns of bilingual speak- across language pairs (e.g.. Turkish-Englise FreechhEngglis and Spanish-English) in terms of type and frequency. If this the case, just relying on the self-dcclarations of the speakers/users about their language backgrounds and collecting random CSW data will not represent the variation between multilingual speakers/users with varying degrees of language proficiency.
Age: Reyes (2004) explains how functions of CSW differ between the two groups of bilingual (Spanish-English) participants belonging to differ- ent age groups. Similarly, Ellison and Si (2021) find significant differences in terms of CSW pat- terns between the older and younger bilingual (Hindi-Ennge Ennndd) speakers in India. Considering the evidence for age related CSW variation, limiting the data collection to certain age groups (c.g., only university students) may not represent the CSW patterns for different age groups (e.g., youngsters and/or elderly) in the same population.
As illustrated with literature above, there is a clear link between the socio-demographic factors and CSW. Without knowing the socio-demographic information about the speakers/users in a CSW data set, it is not possible to assess what type of CSW patterms represent which type of speak- ers/users and/or the variation among them. Any type of CSW data collected without taking the socio-demographic information about the speak- ers/users into account will face the risk of under- representing or overrepresenting certain groups in the target multilingual population.
Gender: Finnis (2014) explores the role of gender and identity on the CSW (English-Greek) within the Greek-Cypriot community in London highlighting the differences between male and female bilingual speakers in terms of CSW pat- terns. Similarly, Farida et al. (2018) finds link between CSW and gender identity marking for Urdu-English bilingual women while Gulzar et al. (2013) indicate differences in CSW patterns (Urdu- English) male and female teachers in terms of CSW patterns during classroom communication in Pak- istan, Agarwal et al. (2017) indicate gender dif- ferences for using offensive language within the Hindi- Hindi-English CSW social media data set. Con- sidering the evidence for gender related variation in CSW patterns across language pairs, there is a need collect more representative CSW data sets reflecting CSW use by both genders in the target populations.
2.4.2 Data Collectors, Transcribers and Annotators
Who collects, transcribes and annotates the data is as important and who produces it. In that sense, Prabhakaran et al. (2021) suggest that the socio- demographic backgrounds of the annotators should represent and align with the diversity in society to prevent biases toward certain groups or individu- als. Similarly, Sap et al. (2022) find link between how annotators perceive toxicity based on their socio-demographic profiles and beliefs. In terms of collecting and preparing the CSW data, is crucially important to recruit data collectors, tran- scribers and annotators who are representative of the multilingual target population and/or who are
Language Background: Language back- grounds of the speakers/users are often taken for granted while collecting CSW data. First of all, most CSW data sets focus on certain language pairs ignoring the fact that the same speakers could also speak other languages in their daily lives. For ex-

---

aware of the cultural and social dynamics in the multilingual community. Below, we discuss the importance of socio-demographic factors for the CSW data preparation team as follows.
of annotation errors and system failure eventu- ally. Similar failures could also be observed due limited language proficiency of the data collectors, transcribers and annotators in the CSW data sets as well.
Gender: Nortier (2008) hired male assistant to collect CSW speech data among the male and bilin- members of the Morroc- can immigrant community (Netherlands) to make them comfortable about the data collection process instead of a female assistant. Similarly, Doğruöz and Sitaram (2022b) provide failed examples of language technologies that could not achieve col- lecting naturalistic data in a rural setting in India since the female speakers were reserved about talk- ing to (male) data collectors and they could not talk naturally in presence of their elderly.
Age: Student populations are convenient sam- ples for transcription and annotation tasks for con- sidering the limited time and resources. However, Al Kuwally et al. (2020) show that age of the anno- tators influences the annotation task in hand. In that sense, limiting the age of the CSW data preparation team members university students may face is- sues about representativeness considering the vari- ation between CSW patterns and age (discussed in section 2.4.1). As a fresh perspective, Nekoto et al. (2022) recommend involving the members from the community and training them for the annota- tion tasks to prevent the representativeness issues across different age groups
gual (e.g.,
In terms annotators, Al Kuwatly et al. (2020) did not observe a link between the gender of the an- notators and bias toward the task in hand whereas Binns et al. (2017) observed differences between males and females annotators while annotating of- fensive content. Although it is not reported ex- plicitly, similar concerns may hold true for data collectors, transcribers and annotators who work on CSW data with offensive content (e.g., Agarwal et al. (2017) on Hindi-English CSW data set on offensive language). To achieve representativeness in the preparation of CSW data sets, there is a need to report the actual practices about the gender bal- ance in data collection, transcription and annotation teams.
More recently, generative Al models are used to label the data and these models may even surpass the accuracy of human annotations (e.g., He et al. (2023), Wei et al. (2022), Kuzman et al. (2023)). Considering the low performance of generative models in low resource languages (Ahuja et al., 2023), the benefit of such models for annotating multilingual and CSW data sets is unclear. Until significant improvements in that area, selecting rep- resentative annotators to annotate the CSW data will remain relevant.
Language Background: Claiming to know the languages in the CSW data sets is often enough to be hired as a transcriber and/or as annotator espe- cially when there are not a lot of eligible candidates. However, without proper understanding about the multilingualism in the given context and determin- ing the level of proficiency required for the task can be insufficient to hire representative sam- ple of transcribers and annotators based on their backgrounds. As an evidence for an unrepresenta- tive selection of annotators based on their language backgrounds, Diab (2023) describes a failed ex- ample of hate speech detection system which included multiple dialects of Arabic. The annota- (ors(recruited for this task) were only able to speak the Arabic dialect spoken in Morocco whereas the data set included examples from other Arabic di- alects as well. As a result, recruitment of anno- tators whose language skills do not represent the language/dialect in this task led to a high number
2.5 Data Selection and Filtering
Lack of insights about the additional factors in the multilingual context and/or filtering processes have implications for the performance of systems. For example, Shah al. (2020) build ASR systems using monolingual and CSW data filtered from the same corpus and find that models that perform well on CSW data do not perform well on the monolin- gual data (and vice versa). However, both CSW and monolingual speech co-occur in the original speech data set and they are even spoken by the same speakers. This indicates that creating corpora (whether monolingual or CSW) through filtering (or cleaning) the CSW data randomly (by script or language) leads to issues in the performance of such systems since the new data does not represent the real-life communication where both CSW and monolingual speech stand together.

---

3
Current Practices for CSW Data Collection and Preparation
approaches for collecting CSW speech data as fol- lows: 1) Utilizing already available CSW data (and their transcriptions) in different formats (e.g., meet- ing recordings, news broadcasts on TV and radio, entertainment shows (e.g., soap operas, TV) and parliamentary debates). II) Speakers were asked to read aloud prompts (containing CSW) which were generated by scraping written text from web pages and they are processed further create phoneti- cally balanced prompts. III) Speakers were asked to talk about certain topies that may elicit CSW during an informal conversation which was simul- taneously recorded and transcribed afterwards. IV) Speakers were asked to converse with an automated system that code-switches. Majority of the CSW speech data sets (51 %) were collected according to the third approach.
To have a better understanding about the data col- lection and preparation procedures, we analyzed CSW research published in ACL Anthology and Interspeech between 2008-2023. We mainly fo- cus on the publications describing CSW data sets that make use of speaker and user generated data (i.e., speech data, social media posts) and exclude the ones based on written and historical sources (e.g., Liu and Smith (2020)) and non-user gener- ated content (c.g., movie scripts and information retrieval data as in Sequiera et (2015), Mehnaz et al. (2021), Khanuja et al. (2020a), Chandu et al. (2019), Raghavi et al. (2015), Pratapa and Choud- hury (2017b)). If the same data set (e.g., van der Westhuizen and Niesler (2018)) was used in other related studies multiple times (e.g.. Biswas et al. and Wilkinson et al. (2020)), we only report the reference associated with the original data set.
3.2 CSW Social Media Data Sets
We surveyed 27 studies (see Table 3 in Appendix) which released 39 user-generated and CSW textual data (i.e., social media) sets. Two techniques used for creating CSW social media data sets were I) scraping the data from existing posts/comments from Youtube WhatsApp chats and blogs (by look- ing for specific keywords or topics that are likely to contain CSW and II) using chat data from multi- lingual users who were instructed to code-switch. Majority (97%) of the these data sets were collected according to the first approach.
We are also aware of the artificially created CSW data sets which are derived from user-generan user-generated data through translation (c.g.. Duong et al. (2017); Nakayama et al. (2018); Banerjee et al.; Mehnaz et al. (2021); Gupta et al. (2018); Winata et al. (2019)). However, it is not always clear how the data is translated into CSW in these cases (e.g.. Jayarao and Srivastava (2018)). Additionally, is possible to generate CSW text data synthetically, either by using computational implementations of linguistic theories to generate data from monolin- gual sentences (e.g., Pratapa et al. (2018), Li and Fung (2014), Tarunesh et al. (2021)) or learning patterns from real user-generated CSW data (c.g., Garg et al. (2018)). However, synthetic data pro- vides diminishing returns when used in models that are already trained on real CSW data (Khanuja et al., 2020b). In sum, more research needed to determine what kind of complementary infor- mation synthetic CSW data provides to linguistic models in comparison to user ger-ggenrateed CSW data in real-life communication contexts. Therefore, practices around artificial CSW data sets are also excluded from our study.
4 Results
In this section, we explain the results of our find- ings for the data sets we have surveyed based on the quality check and representativeness criteria presented in section (2).
4.1 Location Based Variation in Data Collection
CSW speech data sets were collected from loca- tions all around the world (e.g. 13% from EU, 20% from South East Asia, .03% from Australia, 27% from Africa, .06% from USA, 13% from India, .06% from an International Meeting, .06% through crowdsourcing without any location specification). Despite the variation in terms of location, there are still issues about representativeness covering the variation in respective language pairs. For example, the Spanish-Englin data sets in the US were col- lected from the bilingual speakers who came from Mexico. However, there are also Opanish-Englise
3.1 CSW Speech Data Sets
Automatic Speech Recognition (ASR) systems are typically trained on large volumes of transcribed speech data. We have surveyed 25 papers that re- leased 29 CSW speech data sets (see Table 2 in Ap- pendix). Based on these papers, we identified four

---

Due to the nature of the social media data, the location where CSW data sets in Table 3 (in ap- pendix) are constructed is not known. Without this information, it is not even possible discuss is- bilinguals in the US from other Spanish speaking countries (e.g., Puerto Rico and Cuba). Similar to example (1) presented in section 2.1., there could also be differences in the CSW patterns of these communities but they are not represented in the current Enngish CSW data sets.
skills or systematic self-declarations of the speak- ers. Some studies (e.g., (Hamed al., 2018), (Çetinoğlu and Çöltekin, 2022), (Lyu et al., 2010)) recruited multilingual university students whose CSW may not represent the CSW in general pop- ulation (cf. section 2.4). None of the CSW data sets reported the gender or age of the transcribers and only .06% of the data sets reported the bare minimum about the language backgrounds of the transcribers based on the subjective impressions of the researchers.
sues about representativeness for the current CSW social media data sets.
CSW Social Media Data Sets: Except Bar- man et al. (2014), the age, gender and the lan- guage backgrounds of the users were not deseribed in any of the CSW social media data sets (Ta- ble 3). The age and gender of the annotators were mentioned in only one study (Chakravarthi et al. (2020b)) whereas the language backgrounds were mentioned in three studies (i.e., Jamatia et al. (2016), Chakravarthi et al. (2020a), Barman et al. (2014)). However, this information was not de- scribed systematically and the language skills of the annotators were reported based on the subjec- tive impressions of the researchers which may not represent the real-life linguistic performances of the annotators. Lack of socio-demographic infor- mation about the users and the annotators make it difficult to assess the representativeness of their CSW patterns in these data sets in comparison to the general population.
4.2 Coverage of Languages
In terms of linguistic diversity, 72% of CSW speech data sets involved language pairs consisting of En- glish and another language (e.g., Hindi, Mandarin, Vietnamese, South African languages, Spanish). The rest of the CSW speech data sets included different language pairs (e.g., Ukranian-Russian, German-Turkinn Frisian-Dutch Modern Standard Arabic-Arabic Dialects and Arabic French). 92% of CSW social media data sets involved English as one of the language pairs and 28% were about Hindi-English CSW. Considering many other CSW language pairs/tuples spoken around the world by millions of speakers, there is urgent need to in- crease CSW data sets representing other language pairs/uples.
4.3 Registers
Within the CSW speech data sets, 72% involved informal register whereas 97% of the CSW social media data was in informal register across all lan- guage pairs. In that sense, CSW in formal registers is currently underrepresented in both speech and social media data sets.
4.5 Filtering
CSW Speech Data: While eliciting CSW speech data interviews informal conversations from multilingual speakers, it is important document the selection criteria for the conversational topics and the exact instructions given to multilingual speakers to assess whether the linguistic output represents the real-life language use for the same group. However, this type of explanation or justi- fication was mostly lacking among the CSW data sets have surveyed (except Nguyen and Bryant (2020b), Lovenia al. (2021)).
4.4 Socio-demographic Variation
CSW Speech Data Sets: Except Nguyen and Bryant (2020a), criteria for selecting data collec- tors, transcribers and annotators were not men- tioned explicitly in any of the reviewed data sets. Table (2) illustrates the CSW speech data sets in this study and 44% of these studies mentioned the gender, 36% mentioned the age and 52% included some information about the language backgrounds of the speakers. Except Lovenia et al. (2021) and Hamed et al. (2020), descriptions about the language backgrounds of the speakers were lim- ited to subjective impressions of the researchers rather than objective measurements about language
For example, Sivasankaran et al. (2018) asked the participants to have informal conversations on pets and relationships. However, the reasons for selecting these particular topics (instead of others) were not described. Lack of explanation about the selection of topics for conversations hampers the representativeness of these data sets in comparison to real-life communication.

---

Among the CSW social media data sets we have reviewed, the data was collected through scraping different types of social media posts (e.g. FB, Twit- ter) and through manual search using a list of key- words (e.g. on YT videos). However, none of the data sets included description about how these de- cisions were taken (e.g, choice of certain keywords hashtags instead of others).
multilingual communication is valuable and it will not serve the needs and preferences of the mul- tilingual users/speakers. To build sustainable and reliable CSW systems, there is a need to consider representativeness as the key issue for collecting and preparing CSW data for further processing.
The need for data statements and/or guidelines in computational linguistics (Bender and Friedman (2018); Gebru al. (2018)) and machine learning (Geiger et al., 2020) have been voiced earlier due to ethical and bias concern. Although we acknowl- edge them, there are also issues specific to repre- sentativeness in data collection and preparation in multilingual communication. Instead of creating a new list of guidelines, we present the reader with compact check-list to consider when they are col- lecting and preparing representative CSW data sets as follows:
CSW datasets are often constructed by filtering a larger data set which has also monolingual parts If the languages in the CSW data set have different scripts, a filtering could be applied by selecting one of the scripts. However, this practice raises issues about representativennss of CSW data in real-life settings. For example, Srivastava and Singh (2020) and Patro al. (2017) filter sentences in Hindi- English data so that they only contain the Latin script but this practice leads to a loss of valuable CSW data written in the Devanagari script or a mix of scripts (c.g., Latin and Devanagari). So far, we did not come across enough information aon what is filtered in CSW data sets (e.g., the exact keywords, the criteria about what counts a borrowing vs. CSW, topics of discussion), what is left out, how much data is lost during the filtering and the implications of such filtering on the real- life usage of a system built with such filtered data (e.g., what type of errors are prevented or created by such filtering on the data). Therefore, it is hard to assess the representativeness of these multilingual and CSW social media data sets in comparison to real-life language use for these communities.
How the location based linguistic variation represented in the CSW data set?
How is the register based variation represented in the CSW data set?
How is the socio-demographic variation in the multilingual community is represented at the data collection (for speakers/users, data collec- tors) and preparations stages (for transcribers and/or annotators)?
Which data filtering procedures were applied during the data preparation stage and how do these procedures influence the representative- ness of the CSW data in comparison to real- life communication?
5 Discussion and Conclusion
Considering data as the backbone of language tech- nologies, our goal was to investigate the reasons behind the lack of progress in CSW language tech- nologies through documenting the current data col- lection and preparation procedures. In line with this goal, we reviewed 52 studies and 68 data CSW data sets in speech and social media domains in terms of representativeness in terms of location based variation, coverage of languages, register and socio- verii-ddmooraphicc variation and filtering prac- tices. Our results indicate that current practices around data collection and preparation in CSW are far from reporting the rationale behind the choices and procedures systematically, Despite the increas- ing capacities and performances of MMLMs, the forgotten lesson is that abundance of CSW data which does not represent the variation real-life
Although there are many different factors con- tributing to the success and failure of systems lan- guage technologies, following the above mentioned check-list list and our detailed explanations about these items in the previous sections of this paper will im- prove the representativeness of data collection and preparation stages for the forthcoming CSW data sets.
Limitations
Although multilingual language use also manifests itself in underlying levels (e.g.. grammatical influ- ences across languages in Thomason and Kauf- man (1992), Bakker and Mous (2013)) Doğruöz and Backus (2009), Doğruöz and Nakov (2014)) mostly surface level features (i.e., CSW) have been

---

studied over the past 16 years in computational linguisties. Therefore, we limit ourselves to CSW research for this paper. Our study is limited to sci- entific publications and we do not have visibility over industrial practices and/or applications about
Ahmed Ali, Shammur Chowdhury, Amir Hussein, and Yasser Hifuy. 2021. Arabic swithshit speech recognition using monolingual data, arXiv preprint arXiv:2107.01573.
D Amazouz, M Adda-Decket and I. Lamel. 2016 frencc-frenchh code- switching across maghreb ara- bic dialects: a quantitative analysis. In Workshop "Corpus-driven studies of heterogeneous and multi- lingual corpora. pages 5-7.
the topics we address in this paper.
Ethics Statement
Our study draws conclusions based on existing lit- erature, and our empirical work explored the effects of regional differences on code-switching. Our pa- per highlights the open questions, major obstacles. and unresolved issues in multilingualism and code- switching in Computational Linguistics.
Djegdjiga Amazouz, Martine Adda-Decket and Lori Lamel. 2017. Addressing code-switching in french/algerian arabic speech. In Interspeech 2017, pages 62-66.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449 222244244600 12460.
References
Peter Bakker and Maarten Mous. 2013. Mixed lan- guages. Oxford University Press.
Usage statistics for content languages for web- sites. https://w3techh com//technologies/ overview/content_anggggg Accessed: 2023-01- 20.
Somnath Banerjee, Sudip Kumar Naskar, Paolo Rosso, and Sivaji Bandyopadhyay. The first cross-script code-mixed question answering corpus.
Prabhat Agarwal, Ashish Sharma, Jeenu Grover, Mayank Sikka, Koustav Rudra, and Monojit Choud- hury. 2017. may talk in english but gaali toh hindi mein hi denge: A study of english-hindi code- switching and swearing pattern on social networks. In 2017 9th International Conference on Communi- cation Systems and Networks (COMSNETS), pages 554-
Utsab Barman, Amitava Das, Joachim Wagner, and Jen- nifer Foster. 2014. Code mixing: challenge for language identification in the language of social me- dia. In Proceedings of the first workshop on compu- tational approaches to code switching, pages 13-23.
Rafiya Begum, Kalika Bali, Monojit Choudhury, Kous- tav Rudra, and Niloy Ganguly. 2016. Functions of code-switching in tweets: An annotation frame- work and some initial experiments. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC" 16), pages 1644 1650.
Basem HA Ahmed and Tien-Ping Tan. 2012. Automatic speech recognition of code switching speech using 1- best rescoring. In 2012 International Conference on Asian Language Processing, pages 137-140. IEEE
Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint 200.:2228..22..
Emiil M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics. 6:587-604.
Alham Fikri Aji. Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Ma- hendra, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Timothy Baldwin, Jey Han Lau, and Sebastian Ruder. 2022. One country, 700+ lan- guages: NI.P challenges for underrepresented lan- guages and dialects in Indonesia. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7226-7249, Dublin, Ireland. Association for Computational Linguistics.
Irshad Bhat, Riyaz Ahmad Bhat, Manish Shrivastava, and Dipti Misra Sharma. 2017. Joining hands: Ex- ploiting monolingual treebanks for parsing of code- mixing data. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. pages 324-330.
Douglas Biber. 1993. Representativeness in corpus de- sign. Literary and linguistic computing. 8(4):243- 257.
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop Online Abuse and Harms, pages 184-190, Online. Association for Computational Linguistics.
Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. 2017. Like trainer, like bot? in- heritance of bias in algorithmic content moderation. In Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Proceedings, Part HI 9, pages 405-415. Springer.

---

Astik Biswas. Febe de Wet, Ewald van der Westhuizen, and Thomas Niesler. supervise acoustic and language model training for english-isizulu code- switched speech recognition. In LREC 2020 Work- shop Language Resources and Evaluation Confer- ence 11-16 May 2020, page 52.
Margaret Deuchar. 2008. The miami corpus: Doc- umentation file. Bangortalk, bangortalk. org. uk/docs/Miami doc. pdf.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
Tomas Borovicka, Marcel Jirina Jr, Pavel Kordik, and Marcel Jirina. 2012. Selecting representative data sets, Advances in data mining knowledge discovery and applications, 12:43-70.
Özlem Çetinoğlu and Çağrı Çöltekin. 2022. Two lan- guages, one treebank: building a turkish-german code-switching treebank and its challenges. Lan- guage Resources and Evaluation, pages 1-35.
Anik Dey and Pascale Fung. 2014. A hindi-english code-switching corpus. In LREC, pages 2410-2413
Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John P McCrae. 2020a. Corpus creation for sentiment analysis in code-mixed tamil-english text. LREC 2020 Workshop Language Resources and Evaluation Conference 1 11-16 May 2020, page 202.
Mona Diab. 2023. Towards a responsible nip: Walking the walk, emnlp 22 keynote speech.
Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fa- had AlGhamdi, Nada Almarwani, and Mohamed Al- Badrashiny. 2016. Creating a large multi- multi-layered rep- resentational repository of linguistic code switched arabic data. In Proceedings of the Temth International Conference on Language Resources and Evaluation (LREC"16), pages
Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020b. Corpus creation for sentiment amalysis in code-mixed text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collabora- tion and Computing for Snder-Resourced Languages (CCURL). pages 202-210, Marseille, France. Euro- pean Language Resources association.
A. Seza Doğruöz and Preslav Nakov. 2014. Predicting dialect variation in immigrant contexts using light verb constructions. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLPP), pages 1391-1395, Doha, Qatar. Association for Computational Linguistics.
Khyathi Chandu, Ekaterina Loginova, Vishal Gupta, Josef van Genabith, Günter Neumann, Manoj Chin- nakotla, Eric Nyberg, and Alan W Black. 2019. Code-mixed question answering challenge: Crowd- sourcing data and techniques. In Third Workshop on Computational Approaches to Linguistic Code- Switching, pages 29-38. Association for Computa- tional Linguistics (ACL).
A. Seza Dogruöz and Ad Backus. 2009. Innovative con- structions in dutch turkish: An assessment of ongoing contact-induced change. Bilingualism: Language and Cognition, 12(1):41-63.
A. Seza Dogruöz and Sunayana Sitaram. 2022a. Lan- guage technologies for low resource languages: soci- olinguistic and multilingual insights. In Proceedings of the Ist Annual Meeting of the ELRA/ISCA Spe- cial Interest Group on RedderRee Languages, pages 92- 22. European Language Resources Associ- ation (ELRA).
Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021. Towards one model to rule all: Multilingual strategy for di- alectal code-switching arabic asr. arXiv preprint
arXiv:2105.144. 14779.
Bernard Comrie, Martin Haspelmath, and Balthasar Bickel. 2008. The leipzig glossing rules: Conven- tions for interlinear morpheme-by by myrrmheee glosses. Department of Linguistics of the Max Planck Institute for Evolutionary Anthropology & the Departentt of Linguistics of the University of Leipzig. Retrieved January, 28:2010.
A. Seza Dogruöz and Sunayana Sitaram. 2022b. Lan- guage technologies for low resource languages: Soci- olinguistic and multilingual insights. In Proceedings of the Special Interest Group on Under-resourced Languages (SIGUL) at Language Resources and Evaluation Conference(LREEC22
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACI.
A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bul- lock, and Almeida Jacqueline Toribio. 2021. A sur- vey of code-switching linguistic and social perspec- tives for language technologies. In 59th Annual Meet- ing of the Association for Computational Linguis- rics and the 11th International Joint Conference on Natural Language Processing, VOL I (ACL-UJCNLP 2021). pages 1654-1666. Association for Computa- tional Linguistics (ACL).
Maya Khemlani David. 2003. Role and functions of code-switching in malaysian courtrooms.

---

Long Duong, Hadi Afshar. Dominique Estival, Glen Pink, Philipp R Cohen, and Mark Johnson. 2017. Mul- tilingual semantic parsing and code-switching. In Proceedings of the 21st Conference on Computa- lional Natural Language Learning (CoNLL 2017), pages 379-389.
Conference on Natural Language Processing, pages 249-258, Varanasi, India. NLP Association of India
Deepak Gupta, Ankit Lamba, Asif Ekbal, and Pushpak Bhattacharyya. 2016b. Opinion mining in a code- mixed environment: A case study with govermment portals. In Proceedings of the 13th International Conference on Natural Language Processing, pages 249-258.
T Mark Ellison and Aung Si. 2021. A quantitative analysis of age-related differences in hindi-english code-switching. International Journal of Bilingual- ism, 25(6):1110-1222
Vishal Gupta, Manoj Chinnakotla, and Manish Shrivas- tava. 2018. Transliteration better than translation? answering code-mixed questions over a knowledge base. In Proceedings of the Third Workshop on Com- putational Approaches to Linguistic Sode-Switching. pages 39-50.
Panhwar Farida, Saima Murtaza Pandhiani, and Ameer Ali Buriro. 2018. Code-switching and gender identity. The Women-Annual Research Journal of Gender Studies, 10(10).
Katerina A Finnis. 2014. Variation within a greek- cypriot community of practice in london: Code- switching, gender, and identity. Language in society, 43(3):287-310.
Injy Hamed, Mohamed Elmahdy, and Slim Abdennad- her. 2018 Collection and analysis of code-switch egyptian arabic-english speech corpus. In Proceed- ings of the Fleeenth International Conference on Lan- guage Resources and Evaluation (LREC 2018).
Juan Carlos Franco and Thamar Solorio. 2007. Baby- steps towards building spanglish language model. In International conference on intelligent text pro- cessing and computational linguistics, pages 75 84. Springer.
Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020. Arzen: A speech corpus for code-switched egyptian arabic-englis.. In Proceedings of the 12th Language Resources and Evaluation Conference. pages 4237-4246.
Bjöm Gambäck and Amitava Das. 2014. On measuring the complexity of code-mixing. In Proceedings of the 11th international conference natural language processing, Goa, India, pages 1-7.
Silvana Hartmann, Monojit Choudhury, and Kalika Bali. 2018. An integrated representation of linguistic and social functions of vode-switching. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Sreeram Ganji, Kunal Dhawan, and Rohit Sinha. 2019, Titg-hingcos corpus: A hinglish code-switching database for automatic speech recognition. Speech Communication, 110:76-89.
Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang. Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annota- tors. arXiv preprint wxXi:2303..688.
Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. 2018. Dual language models for code switched speech recognition. Proc. Interspeech 2018, pages 2598- 2602.
Megan Herrera, Ankit Aich, and Natalie Parde. 2022. Tweettaglish: A dataset for investigating tagalog- english code-switching. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 2090-2097.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, and Kate Crawford. 2018. Datasheets for datasets.
Susan C Herring. 1996. Computer-mediated communi- cation. Computer-Mediated Communication, pages 1-332.
R Stuart Geiger, Kevin Yu, Yanlai Yang. Mindy Dai. Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. Garbage in, garbage out? do machine learning ap- plication papers social computing report where human-labeled training data comes from? In Pro- ceedings of the 2020 Conference Fairness, Ac- countability, and Transparency, pages 325-336.
Anupam Jamatia and Amitava Das. 2016. Task report: Tool contest on pos tagging for code-mixed indian social media (facebook, twitter, and whatsapp) text @ icon 2016.". Proceedings of ICON.
Malik Ajmal Gulzar, Muhammad Umar Farooq, and Muhammad Umer. 2013. Inter-sentential patterns of code-switching; A gender-based investigation of male and female efl teachers. International Educa- tion Studies, 6(1 1444
Anupam Jamatia, Björn Gambäck, and Amitava Das. 2016. Collecting and annotating indian social media code-mixed corpora. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 406-417. Springer.
Deepak Gupta, Ankit Lamba, Asif Ekbal, and Pushpak Bhattacharyya. 2016a. Opinion mining in code- mixed environment: A case study with government portals. In Proceedings of the 13th International
Pratik Jayarao and Aman Srivastava. 2018. Intent detec- tion for code-mix utterances in task oriented dialogue systems. 2018 International Conference on Elec- trical, Electronics, Communication, Computer, and

---

Optimization Techniques (ICEECCOT), pages 583- 587. IEEE
Barcelona, Spain (Online). International Committee on Computational Linguistics.
Olha Kanishcheva, Tetiana Kovalova, Maria Shvedova, and Ruprecht von Waldenfels. 2023. The parlia- mentary code-switching corpus: Bilingualism in the ukrainian parliament in the 1990s-2020s. In Pro- ceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), pages 79-90.
Holy Lovenia, Samuel Cahyawijaya. Genta Indra Winata, Peng Xu, Xu Yan. Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J Barezi, al. 2021. Ascend: A spontaneous chinese-english dataset for sode-switching in multi-turn conversation. arXiv preprint 122.223
Emaliana Kasmuri and Halizah Basiron. 2019. Building a malay-english code-switching subjectivity corpus for sentiment analysis. Int. J. Advance Soft Compu. Appl. 11(1).
Teresa Lynn and Kevin Scannell. 2019. Code-switching in irish tweets: A preliminary analysis. In Proceed- ings of the Celtic Language Technology Workshop, pages 32-40.
Simran Khanuja. Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020a. A new dataset for natural language inference from code- mixed conversations. Proceedings of The 4th Workshop on Computational Approaches to Code Switching, pages 9-16.
Dau- Cheng Lyu, Tien-Ping Tan, Eng Siong Chng,, and Haizhou Li. 2010. Seame: a mandarin-english code-switching speech corpus in south-east asia. In Eleventh Annual Conference of the International Speech Communication Association.
Suraj Maharjan, Elizabeth Blair, Steven Bethard, and Thamar Solorio. 2015. Developing language-tagged corpora for code-switching tweets. In Proceedings of The 9th Linguistic Annotation Workshop, pages 72-84.
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury. 2020b. Gluecos: An evaluation benchmark for code- switched nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3575-3585
Deepthi Mave, Suraj Maharjan, and Thamar Solorio. 2018. Language identification and analysis of code- switched social media text. In Proceedings of the third workshop on computational approaches to lin- guistic code-switching, pages 51-61.
Didem Koban. 2013. Intra-sentential and inter-
sentential code-switching in turkish-english bilin- guals in new york city, us. Procedia-Socia and Behavioral Sciences, 70: 1 177-
Laiba Mehnaz, Debanjan Mahata, Rakesh Gosangi, Uma Sushmitha Gunturi, Riya Jain, Gauri Gupta, Amardeep Kumar, Isabelle Lee, Anish Acharya, and Rajiv Rain Shah. 2021. Gupshup: An anno- tated corpus for abstractive summarization of open- domain code-switched conversations. arXiv preprint UrXiv:2104.08577
Taja Kuzman, Igor Mozetic, and Nikola I. JuubeKic. 2023. Chatgpt: Beginning of an end of manual linguistic data annotation? use case of automatic genre identifi- cation.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Demoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large lan- guage models in multilingual learning.
Elena Alvarez Mellado and Constantine Lignos. 2022. Borrowing codeswitching? annotating for finer- grained distinctions in language mixing. arXiv preprint arXiv:2206. 04993..
Sophia Lee and Zhongqing Wang. 2015. Emotion in code-switching texts: Corpus construction and analy- sis. In Proceedings of the Eighth SIGHAN workshop on chinese language processing, pages 91-99.
Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. Qasr: Qeri al- jazeera speech resource a large scale annotated arabic speech corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2274-2285.
Ying Li and Pascale Fung. 2014. Language modeling with functional head constraint for code switching speech recognition. In Proceedings of the 2014 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 907- 906-916.
Sahoko Nakayama, Takatomo Kano, Quoe Truong Do, Sakriani Sakti, and Satoshi Nakamura. 2018. Japanese-english code-switching speech data con- struction. In 2018 Oriental COCCOSDA-nternation Conference on Speech Datahase and Assessments, pages 67-71. IEEE.
Ying Li, Yue Yu, and Pascale Fung. 2012. mandarin- english code-switching corpus. In Proceedings of the Eighth International Conference Language Resources and Evaluation (LREC" 12), pages 2515- 2519.
Shijia Liu and David Smith. 2020. Detecting de min- imis code-switching in historical German books. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1814-1814,
Roberto Navigli, Simone Conia, and Bjöm Ross. 2023. Biases in large language models: Origins, inventory and discussion. J. Data and Information Quality. Just Accepted.

---

Wilhelmina Nekoto, Julia Kreutzer, Jenalea Rajab, Mil- licent Ochieng, and Jade Abbott. 2022. Participatory translations of oshiwambo: Towards sustainable cul- ture preservation with language technology. In 3rd Workshop on African Natural Language Processing.
of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2264-2274
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing amnotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bee Chin Ng and Francesco Cavallaro. 2019. Multilin- gualism in southeast asia: The post cost-colonial language stories of hong kong, malaysia and singapore. Multi- disciplinary Perspectives on Multilingualism, pages 27-50.
Adithya Pratapa, Gayatri Bhat. Monojit Choudhury. Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1543-1553.
Dong Nguyen and A Seza Dogruöz. 2013. Word level language identification in online multilingual com- munication. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 857-862.
Dong Nguyen, A. Seza Dogruöz, Carolyn P. Rosé, and Franciska de Jong. 2016. Computational sociolin- guistics: : a survey. COMPUTATIONAL LINGUIS- TICS, 42(3):537-593.
Adithya Pratapa and Monojit Choudhury. 2017a. Quan- titative characterization of code switching patterns in complex pulti-party conversations: case study on Hindi movie scripts. In Proceedings of the 14th International Conference on Natural Language Pro- cessing (ICON-2017), pages 75-84, Kolkata, India NLP Association of India.
Li Nguyen and Christopher Bryant. 2020a. Can the canberra Vietnamese-English code-switching nat- ural speech corpus. In Proceedings of the 12th Lan- guage Resources and Evaluation Conference, pages $121-4129, Marseille, France. European Language Resources Association.
Adithya Pratapa and Monojit Choudhury, 2017b. Quan- titative characterization of code switching patterns in complex multi-party conversations: case study on Hindi movie scripts. In Proceedings of the 14th International Conference on Natural Language Pro- cessing (ICON-2017), pages 75-84, Kolkata, India NI.P Association of India.
Li Nguyen and Christopher Bryant. 2020b. Canvec-the canberra vietnamese-english code-switching natural speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4121 4129.
Jacomine Nortier. 2008. Types and sources of bilingual data. The Blackwell guide to research methods in
Frin Quirk. 2021, Interspeaker code-switching use school- aged bilinguals and its relation with affective factors and language proficiency. Applied Psycholin- guistics, 42(2):367-393.
bilingualism and multilingualism, pages 52.52.
Doreen Osmelak and Shuly Wintner. 2023. The
denglisch corpus of german-english code code-switching. In Proceedingg of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 42-51.
Ella Rabinovich, Masih Sultani, and Suzanne Stevenson. 2019. Codeswitch-redd reddit: Exploration of written multilingual discourse in online discussion forums. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP P) pages 4776-4786.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Khyathi Chandu Raghavi, Manoj Kumar Chinnakotla, and Manish Shrivastava. 2015. " answer ka type kya he?" learning classify questions in code-mixed language. In Proceedings of the 24th International Conference on World Wide Web, pages 853-858.
Ayushi Pandey, Brij Mohan Lai Srivastava, and Suryakanth V Gangashetty. 2017. Adapting monolin- gual resources for code-mixed hindi-enn mondi--eglihhhhh speech recognition. 2017 International Conference on Asian Language Processing (IALP), pages 218-221. IEEE.
Vikram Ramanarayanan and David Suendermann-Oes 2017. Jee haan, i'd like both, por favor: Elicita- tion of code-switched corpus hindi-english and spanish-english numan-machine dialog. In Inter- speech, pages 47-51.
Jasabanta Patro, Bidisha Samanta, Saurabh Singh, Ab- hipsa Basu, Prithwish Mukherjee. Monojit Choud- hury, and Animesh Mukherjee. 2017. All that is english may be hindi: Enhancing language identi- fication through automatic ranking of the likeliness of word borrowing in social media. In Proceedings
Iliana Reyes. 2004. Functions of code switching in schoolchildren's conversations. Bilingual research journal, 28( 1777998.

---

Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection, In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
Thamar Solorio, Elizabeth Blair, Suraj Mahar- jan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop Computational Approaches to Code Switching, pages 62-72.
Thamar Solorio and Yang Liu. 2008. Learning to predict code-switching points. In Proceedings of the 2008 Conference on Empirical Methods in Natural Lan- guage Processing, pages 973-981, Honolulu, Hawaii, Association for Computational Linguisties.
Salim Sazzed. 2021. Abusive content detection in transliterated bengali-englin social media corpus. In Proceedings of the Fifth Workshop Computational Approaches to Linguistic Code-Switching, pages 125-
130.
Vivek Srivastava and Mayank Singh. 2020, Phine: A parallel hinglish social media code-mixed corpus for machine translation. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 41-49.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana ITi,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint urXiv:2211.05700.
Ishan Tarunesh, Syamantak Kumar, and Preethi Jyothi. 2021. From machine translation code-switching: Generating high-quality code-switched text. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 3154- 3169.
Royal Sequiera, Monojit Choudhury, Parth Gupta, Paolo Rosso, et al. 2015. Overview of fire-2015 shared task on mixed script information retrieval.
Sanket Shah, Basil Abraham, Sunayana Sitaram, Vikas Joshi, et al. 2020. Learning ognize code-switched speech without forgetting monolingual speech recognition. arXiv preprint 7rxiv:2006.0078:
Sarah Grey Thomason and Terrence Kaufman. 1992. Language contact, creolization, and genetic linguis- rics. Univ of Californaa Press.
Han-Ping Shen, Chung-Hsien -ssin Wu, Yan-Ting Yang, and Chun-Shan Hsu. 2011. Cecos: A chinese-english code-switching speech database. In 2011 Interna- tional Conference on Speech Database and Assess- ments (Oriental COCOSDA), pages 120-123. IEEE.
Ewald van der Westhuizen and Thomas Niesler. 2018. A first south african corpus of multilingual code- switched soap opera speech. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018).
Prajwol Shrestha. 2014. Incremental n-gram approach for language identification in code-switched text. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 133-138.
Deepanshu Vijay, Aditya Bohra, Vinay Singh, Syed Sar- faraz Akhtar, and Manish Shrivastava. 2018. Cor- pus creation and emotion prediction for hindi-english code-mixed social media text. In Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: student research workshop, pages 128-
Kushagra Singh, Indira Sen, and Ponnurangam Ku- maraguru. 2018. Language identification and named entity recognition in hinglish code mixed tweets. In Proceedings of ACI. 2018, Student Research Work- shop, pages 52-58.
Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014. Pos tagging of english-hindi code-mixed social media content. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 979-979
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Kr- ishna Rallabandi, and Alan W Black. 2020. A survey of code-switched speech and language processing.
Sunit Sivasankaran, Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, and Monojit Choud- hury. 2018. Phone merging for code-switched speech recognition. In Third Workshop on Computational Approaches Linguistic Code Code-switching.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Erin Smolak. Stephanie De Anda, Bianka Enriquez. Diane Poulin-Dubois, and Margaret Friend. 2020. Code-switching in young bilingual toddlers: A longi- tudinal, cross-language investigation. Bilingualism: Language and Cognition, 23(3):500-518.
Nick Wilkinson, Astik Biswas, Emre Yilmaz, Febe De Wet, Thomas Niesler, et al. 2020. Semi- supervised acoustic modelling for five lingull code- switched asr using automatically-segmentt soap

---

opera speech. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourcdd languages (SLTU) and Collabora- tion and Computing for Resoureed Languages (CCURL), pages 70-78.
A Experimental Setup for Table
To investigate the impact regional differences of CSW on (ASR) performance, we trained ASR models on two different Mandarin-English CSW speech datasets, namely ASCEND (Lovenia et al., 2021) and SEAME (Lyu et al., 2010). We utilized the wav2vec 2.0 model (Baevski et al., 2020), pre- trained on English and Chinese corpus of Common Voice respectively, as our ASR models. We used the development split of SEAME as the test dataset, and partitioned the remaining SEAME data into train and development splits with a ratio of 80:20. We followed Lovenia et al. (2021) and used their codes for the experimental and hyperparameters setup. After fine- fine-tuning the ASR models, eval- uated them on both ASCEND and SEAME test splits.
Genta Indra Winata, Alham Fikri Aji, Zheng-Xin Yong, and Thamar Solorio. 2022. The decades progress on sodt-swit research nlp: systematic survey on trends and challenges.
Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from par- allel sentences. In Proceedings of the 23rd Confer- ence Computational Natural Language Learning (CoNLL), pages 271-280.
Brian Yang. 2019, What are youtube's most popular languages.
Zeynep Yirmibeçoglu and Gülşen Eryigit. 2018. Detect- ing code- code-switching between turkish-english language pair. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 110-115.
Zheng-Xo Xin Yong. Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Samuel Cahyawijaya, Holy Lovenia, Lintang Sutawika, Jan Christian Blaise Cruz, Long Phan, Yin Lin Tan, al. 2023. Prompting large language models to generate code-mixed texts: The case of south east asian languages. arXiv preprint arXiv:2303. 13592.
Emre Yilmaz, Adem Derinel, Kun Zhou, Henk den Heuvel, Niko Brummer, Haizhou Li, and David van Leeuwen. 2019. Lcale Speaker Diarization of Radio Broadcast Archives. In Proc. Interspeech 2019, pages 411-415.
Ruochen Zhang, Samuel Cahyawijaya, Jan Chris- tian Blaise Cruz, and Alham Fikri Aji. 2023. Mul- tilingual large language models (yet) code- switchers, arXiv preprint arXiv:230S. 14235.

---

Langunge Backgrounds
Transcribers
Age
Gender
Language Bac Backgrondss
Speakers
Age
Gender
Table 2: CSW on Speech Data Sets
Arabie Speaking Countries
Algeria, Tontsia, Morocco
International Mezting
International Meeting
Location
Australia
Gemmany
Hong Kong
USA
Crowdsourced
India
Hong Kong
Malaysia sia
Egype
France
Singapure
Bgypt
Netherlands
South Africa
Hong Kong
India
Taiwan
Ukruine
USA
India
India
Modem Standard Arabic Arabic Dialects
English Angglis-isiZaan Zolu, Englishh isiXhosa,
English- SettiisnSSessaan.. English: English-Sesotho Sesotho
Hindi- Hindi-Einghn Eingishh Spanish-Enggiie
Algerian Arabic French
Arabie Dialects- Meett--French French
CSW Languages
Vietnanese- English
Gemman Yurkiihh
Mandarin-Ean Ennglsh
English
Spanish- English
Hindi-Bn Hindi-Englie Enggishh
Hindi- Einlish
Malay Enngish
Arabic Arabic-English English
Arabic Arabis-Irrent Irrench
Mandarin- Linglish
Arabiic English
Frisian -utth
Mandarin Eng lighish
Hindi-Enghh English
Chinese: English
Ukronian- Russiun
Spanish- English
Hindi- Lingishh
Hindi-Ba Hindi-Englan -English
Arabic English
Rumanarayanan and Suendermunn- (2017)
van der Westhuizen Niester (2018)
Çetinogla and Çhitekin (2022)
Nguyen and Bryant (2020b)
Francn and Solorio (2007)
Sivasankaran et (2018)
Kumishchova al. (2023)
Chowdhury al. (2021)
Dey and Fung (2014)
Ahmed and Tan (2012)
Mubarak et al (2021)
Amazonz et al. (2016)
Amazouz et al. (2017)
Lovenia (2021)
Hurtmenn et ul. (2018)
Hamed et al. (2018)
Hamed et al. (2020)
Yilmaz et al (2019)
Shen et al. (2011)
Pandey et al. (2017)
Ganji et al. (2019)
Li et (2012)
Lyu al (2010)
Deuchar (2008)
Ali Ati al (2021)
Reference

---

Lang. Background
Annotators
Age
Gender
Lang. Back Background
Users
Age
Gender
Table 3: CSW Social Media Data Sets
Twitter, Comments Online Platform Comments on an Online Platform
Song Lyrics, Movie Reviews,
Arabic Online Commentary Set
Govermment Portal Comments
Astrology documents
and BBC pages
Twitter, Whatsapp
Twitter, Treebank
Youtuhe Comments
Youtube Comments
Roddit Comments
comments
HB Comments
Data Type
FB, Twitter
Twitter, FB
Twitter
Twitter
Twiter
Twitter
Twitter
Twitter
Twittrr
Blogs
Twitter
Weibo
Twitter
Reddit
Twitter
English Hindi, English BB-ggai
Hindi- English. Bengali- English
Spanish- Spanish-Englist English
Eng- Romanian, Eng- Indonesian,
Eng Tagalog, Eng- Greek.
Eng Russian. Ein- Enn- Spanish,
Eng. Turkish. Eng- Arabic, Eng- Eng-Croon Croation, Eng- Albanian Hindi Hindi- Lingissh
MSA Egyptian ptian Arabic
Languages
Spanish Einglish
Bengali- English
Spanish- English
Turkish- Fnnglish
Turkish- Dutch
Hindi- English
Hindi- English
Hindi Ennlish
Hindi- English
Hindi-Dr English
Irish- Irish-Englis. English
Spanish- Pmanish-Enggist Enggissh
Malay- English
Tamil Tamil-Engg English
Bengali-Enge English
German- English
Tagalog- Eggalog-Enggiii
Enggiissh
Mandarin--aa English
Hindi Linglish
Nepali English
Hindi Hindi-Beng Bengali-Engginn
Hindi- Eng
Yirmihesoghn and Eryigit (2018)
Nguyen and Dogruüiz (2013)
Kasmuri and Basiron (2019)
Osmelak and Winterr (2023)
Mellado and Lignos (2022)
Chakravarthi et al. (2020la)
Gambäck Das (2014)
Jamatia and Das (2016)
Lynn Scannell (2019)
Rabinovich al. (2019)
Solorio et (2014)
Sequiera et (2015)
Maharjan et al (2015)
Jamatia et al. (2016)
Herrera et al. (2022)
Lee and Wang (2015)
Gupta et al. (2016bb)
Barman et al. (2014)
Begum (2016)
Vijay et al. (2018)
Vyas et al. (2014)
Diab al, (2016)
Bhat et al. (2017)
Singh et al. (2018)
Mave et al. (2018) 18)
Shrestha (2014)
Sazzed (2021)
Paper


Reviews:
[{"review_id": "o4pxQ8BnZO", "reviewer": "Reviewer_65nB", "summary": "", "questions": "A: This paper mentions only one instance of code-switching with more than two languages, namely in the second table in the appendix (Hindi-Bengali-English). Is this because code-switching between more than two languages is underrepresented in code-switching research (i.e. there were no more such datasets that suited the pre-defined selection critera), or was this a focus chosen by the author(s)? For instance, in line 41, the author(s) only speak of language pairs.\n\nB: Have you investigated the needs and preferences of the target multilingual speakers/users, or is this simply an assumption?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and also the questions. \n\nThis is a position paper which explains the \"representativeness problem\" in code-switching research which is crucially important for the advancement of the field but also from ethical perspectives (e.g. diversity & inclusivity) at various stages of code-switched and multilingual data collection, transcription and annotation. In general, there is a need for more empirical evidence across languages. We provide the first empirical evidence in this direction to support our arguments. Even one example shows what is lacking in terms of representativeness. The goal of this article to bring up the representativeness issue and explain what goes wrong with detailed analysis of existing data collection and preparation for code-switching research targeting specifically the audience of this conference who may take the existing data sets on code-switching for granted and focus on modeling without being aware of the issues on representativeness. In addition, the audience will also benefit from extensive literature on representativeness across other areas of research (e.g., linguistics) with our references.  Hopefully, this clarifies our intentions for this paper and clarifies your hesitations. \n\nHere are our replies for your questions:\n\n- Question A. Code-switching between more than two languages are in fact underrepresented (Dogruoz et al., 2021, Genta el al., 2023), and we highlighted language pairs in line 41 because they are some of the most researched code-switched languages. \n\n- Question B. There has been work done on the preferences of multilingual speakers. One example is how multilingual speakers strongly prefer text-based conversational agents that are capable of code-mixing (Bawa et al., 2023). In addition there is also research from linguistics literature on code-switching which shows the needs  and preferences of multilingual users across different settings. The paper by Dogruoz et al. (2021), Dogruoz & Sitaram (2022) and Androutsopolous (2007) indicate these points with clear examples (e.g. multilinguals in Germany) and failed use cases (e.g. in India). We mention the first two references already in the paper. However, we can make them more explicit and add the last reference to the paragraph where we mention the needs and preferences of multilingual speakers/users specifically. Thanks for your time and the pointers. We definetely benefit from them. \n\nReferences:\n\nWinata, G. I., Aji, A. F., Yong, Z. X., & Solorio, T. (2022). The decades progress on code-switching research in NLP: A systematic survey on trends and challenges. arXiv preprint arXiv:2212.09660.\n\nBawa, A., Khadpe, P., Joshi, P., Bali, K., & Choudhury, M. (2020). Do Multilingual Users Prefer Chat-bots that Code-mix? Let's Nudge and Find Out!. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1), 1-23.\nUpon acceptance, we can add these references to our paper.\n\nDoğruöz, A.S., Sitaram, S., Bullock, B.E., Toribio, A.J. (2021). A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies, ACL, 2021. \n\nDoğruöz, A.S. & Sitaram, S. (2022). Language Technologies for Low Resource Languages: Sociolinguistic and Multilingual Insights. Proceedings of SIGUL at LREC’22. European Language Resources Association.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author(s) critically assess previous practices in data collection and preparation in code-switching corpora, highlighting issues that could be avoided in future work.\n2. The author(s) provide a concrete checklist that could encourage more representative data collection and preparation for code-switching corpora, which the author(s) suggest may help language modelling for code-switching.", "reasons_to_reject": "1. Empirical evidence for the main motivation is limited.\n2. The paper lacks some context in terms of language coverage.\n3. The author(s) assume the 'needs and preferences of the target multilingual speakers/users', but do not explain or motivate these needs or how they were obtained.", "questions_to_authors": "A: This paper mentions only one instance of code-switching with more than two languages, namely in the second table in the appendix (Hindi-Bengali-English). Is this because code-switching between more than two languages is underrepresented in code-switching research (i.e. there were no more such datasets that suited the pre-defined selection critera), or was this a focus chosen by the author(s)? For instance, in line 41, the author(s) only speak of language pairs.\n\nB: Have you investigated the needs and preferences of the target multilingual speakers/users, or is this simply an assumption?", "paper_topic_and_main_contributions": "This paper provides an overview of representativeness in the collection and preparation of corpora of code-switching language. First, the author(s) define representativeness in terms of location, domain (often internet), register, socio-demographics and data selection/filtering. Then, they analyze data collection and preparation in code-switching research along these properties. Apart from this overview, the main contributions of this paper include an experimental result on location-based variation and a concrete list of questions that can be used for ensuring representativeness in future development of code-switching corpora.\n\nThe author(s) critically assess a broad range of research involving the collection and preparation of corpora for code-switching language, including 68 data sets and spanning 15 years of research. This provides a comprehensive view of current practices and clearly highlights the issues when it comes to overlooking representativeness in previous work. They propose a clear and well-defined scope of representativeness and motivate each aspect of their definition with a sufficient number of relevant sources.\n\nIn the introduction, the author(s) frame the need for representative code-switching corpora by arguing that 'the lack of representative CSW data collection and preparation procedures could lead to [the lack of progress in language technology that effectively deals with code-switched language]' (line 43-45). This is illustrated with empirical results in Table 1 for the case of location based variation, where the author(s) compare code-switched Chinese from Hong Kong and Singapore. However, for the other defined dimensions of representativeness, such as register variation and socio-demographic variation, the author(s) assume that the lack of representativeness similarly leads to difficulties in language modelling, but they provide no supporting evidence for this. Even in the case of location based variation, the author(s) only take one language pair in two datasets into account.\n\nThe author(s) assess linguistic diversity by looking at how many of the language pairs/tuples in code-switching corpora include English. They find that 72% of CSW data sets included language pairs consisting of English and another language. However, the meaning of this number is unclear, as the author(s) provide no information regarding the proportion of English in actual code-switching use. This argument would be stronger if there were numbers regarding the actual proportion of code-switching that does not include English. Furthermore, to actually assess linguistic diversity in these corpora, it would be interesing to get information about the included non-English languages and their locations, language families, typological properties, etc.\n\nThe author(s) mention that current data collection and preparation practices in code-switching research 'will not serve the needs and preferences of multilingual users/speakers' (line 698-699). However, it is not clear what these needs exactly are or how they were collected. In the same line of reasoning, the author(s) write that 'Considering many other CSW language pairs/tuples spoken around the world by millions of speakers, there is an urgent need to increase CSW data sets representing other language pairs/tuples for CSW research.' (lines 568-571). This 'urgent need' is not well-motivated: it is not to be assumed that all communities of multilingual speakers benefit from and/or desire language technology research for all (code-switching) scenarios. Thus, it would be helpful if the need for representativeness was supported by actual input from the relevant multilingual communities.\n\nThe properties of representativeness are clearly described and motivated. Lastly, the author(s) provide a concrete checklist for representativeness, which has the potential to help future research be more effective for developing language technology for code-switching users.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vw8kC9yEFI", "reviewer": "Reviewer_sZBB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments about praising our paper and recognizing the importance of representativeness in this under-researched area. About the questions you raise:\n\nWe agree that future work is needed to show how better code-switching data collection practice affects downstream task performance. Nonetheless, our empirical work has pointed out that current practice of code-switching, which treats language pairs as equal without explicitly describing factors of representativeness, is a standing issue as systems trained on them are not necessarily transferable. You mention that other factors (e.g. change in domain, poor labeling, change in accent) may lead to differences in performance of C-S systems. We kindly refer to Shah et al., 2020 paper (which is already in our references) which shows that there is a difference in the performance of speech recognition systems between the accuracies of code-switched and monolingual speech from the same dataset. That is to say, a system containing both code-switched and monolingual speech that is fine-tuned on code-switched speech performs poorly on monolingual data and vice versa. Since these subsets come from the same dataset, other factors such as background noise, recording conditions, speakers etc. remain the same. Although this paper does not show empirical results on different types of code-switching, it does show empirical results on code-switched vs. monolingual speech. We can explain this point in more detail in the camera-ready version of the paper to make the results of this paper more clear.\nAbout your comment on section 2.5: As explained in Dogruoz et al., (2021) and Dogruoz & Sitaram (2022) code-switching is not random and depends on many factors (e.g. topic of conversations, the linguistic ability of the conversational partners, context). It is also quite normal and expected that multilingual speakers do not code-switch all the time but can hold monolingual communication as well. As cited in Papalexakis et al., (2015) for immigrant multilingual contexts, “...multilingual speakers use minority languages to discuss topics related to their ethnic identity and reinforcing intimacy and self-disclosure (e.g. homeland, cultural traditions, joke telling) whereas they use the majority language for sports, education, world politics, science and technology” and they provide examples of studies by Ho, 2007; Androutsopoulos, 2007; Tang et al., 2011. In that sense, it is unrealistic to assume that multilingual speakers switch across languages all the time. We understand that the literature from other areas are usually not visible for researchers in Computational Linguistics. Therefore, we will follow your suggestion and add these references to section 2.5 for the camera-ready version of the paper upon acceptance. Thanks for bringing it up!\n\nDogruoz et al., (2021) and Dogruoz & Sitaram (2022) are already mentioned as references in the current version of the paper. Here are the additional references as evidence for this claim: \n\nJudy Woon Yee Ho. 2007. Code-mixing: Linguistic form and socio-cultural meaning. The International Journal of Language Society and Culture, 21.\n\nDai Tang, Tina Chou, Naomi Drucker, Adi Robertson, William C Smith, and Jeffery T Hancock. 2011. A tale of two languages: strategic self-disclosure via language selection on facebook. In Proceedings of the ACM 2011 conference on Computer supported cooperative work, pages 387–390. ACM.\n\nJannis Androutsopoulos, 2007. The Multilingual Internet, chapter Language choice and code-switching in German-based diasporic web forums, pages 340– 361. Oxford University Press.\n\nPapalexakis, E., Nguyen, D., Doğruöz, A.S. (2014). Predicting code-switching in Multilingual Communication for Immigrant Communities.\n\nIt is correct that the paper is a position paper rather than a survey since we tackle an important issue focusing on the representativeness of data about code-switching. Therefore, we mainly focus on papers which release data sets on code-switching instead of covering all the papers on the same topic. \nTo clarify your first point: Considering that data is essential for LLMs, there is a crucial need to discuss them to highlight our points about the representativeness (or lack of it) for code-switched data sets. Considering the expanding research using LLMs for code-switched data sets (already mentioned with references in our paper), it would have been very ignorant of us not to mention LLMs and their influences on code-switched data sets. \nAbout your second point: We are sorry that you are confused about the structure. We did not hear this comment from the other reviewers. On the contrary, we were praised for the clarity of the paper and our arguments. However, structuring the paper in terms of sections is a matter of personal style which could differ among different authors and/or readers. In any case, we respect your opinion and take it seriously. To clarify, there is a growing and urgent need to define representativeness and explain its importance for code-switching research for computational linguists and other researchers who take representativeness for granted without considering how the data was collected, from who and under which conditions which may lead to biases and excluding groups or communities. These are explained in section (2). Section (3) introduces the code-switched data sets (i.e., speech and social media) building up on the information on section (2). Section (4) focuses on the comparative analyses of these data sets (explained in section 3) based on several criteria (e.g., location based variation, coverage of languages, registers, socio-demographic variation) which contribute to representativeness. In that sense, section (4) is not a mere literature review in the traditional way. Instead, it presents the results of our analyses comparatively. Hopefully, this explanation clarifies the confusion. We can also make the links between these sections more clear upon acceptance. \n\nWe hope our explanations clarify your hesitations and you will consider increasing our score. Thank you for your time and insights in advance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper covers an important yet little-discussed problem in research on code-switching, namely that the datasets are often unrepresentative of the phenomenon they aim to model. \n\n2) It contains an in-depth definition of \"representativeness\" in terms of code-switched data sets.\n\n3) There is an extensive survey of recent research on code-switching published in the ACL Anthology and Interspeech, situated in terms of (a lack of) representativeness in the training/evaluation data.\n\n4) There is some empirical work showing that datasets which claim to be in the same code-switched language actually show significant variation.\n\n5) The checklist on how to collect and prepare representative code-switched datasets is a useful addition", "reasons_to_reject": "1) The main reason to reject this paper is that some of the main claims are not backed up with sufficient analysis and/or citations. In particular, I believe that the main claim of the paper is that a lack of representativeness in training and evaluation data leads to down-stream systems which are not fit for purpose (see lines 85, 694). However, there is no evidence in the paper which backs this up (empirical work here would be really good). Similarly, I liked the empirical work in section 2.1, but I do not think you can draw the conclusion that the changes in code-switching alone cause the difference in performance, unless there is an effort to control for or explain other factors (e.g. change in domain, poor labelling, change in accent). More empirical work and more nuanced conclusions here would be very useful. In section 2.5, there is a claim that current filtering practices for code-switched data degrade performance, but no analysis or citations. On line 643, you make the claim that lack of explanation about the selection of topics for conversations hampers the representativeness of the datasets in comparison with real life conversations, but it is unclear how this fits in with your previous discussion and there is no evidence presented for this point.\n\n\n2) A secondary problem with this paper is that it feels disjointed, so it hard to follow the argument in it (assuming this paper should be understood as a position paper rather than a literature review as suggested in lines 95 to 98). For example, in the introduction, the first paragraphs (to line 46) set out the argument of the paper, but then the following paragraphs read like a general literature review of LLMs, so it is hard to understand the argument of the paper. It would be good to list the contributions of the paper clearly in the introduction and perhaps either shorten the discussion of LLMs or move it to a different section. Sections 2 and 4 are not clearly differentiated: I believe section 2 is supposed to define representativeness and section 4 is supposed to review the literature, but there is currently a lot of review in section 2. This means that section 4 feels repetitive. \n\n\nOverall, I think that this paper presents an important problem and it is clearly well-researched. However, I believe that it requires more empirical work and stronger argumentation to support their claims.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper argues that the current poor performance of systems designed to handle code-switching is due to a lack of representativeness in the training and evaluation data. They explore this hypothesis by defining representativeness in code-switched data and then examining how current work falls short.\n\nMain contributions:\n\n- Critical survey of current work on code-switching in speech and text, focusing on the shortcomings in their training and evaluation datasets\n-  Defining \"representativeness\" in the context of code-switched data, broken down into different aspects\n- Survey of current practices for collecting code-switched datasets\n- Checklist in conclusion to aid more representative code-switched datasets in future\n- Empirical study of performance of ASR model when trained on different datasets in different varieties of the \"same\" language", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hqiRYOPqHG", "reviewer": "Reviewer_5PvD", "summary": "", "questions": "The arguments provided for the evaluation of the analyzed corpora are interesting and allow the observation of several features and respects. \nNevertheless, in my opinion, most of them are not features specific to the code-switching corpora. For instance, the dimensions of variation discussed in the paper can be observed also in corpora where code-switching does not occur. Can the authors better discuss this point and organize the dimensions according to the fact that they refer or not only to code-switching corpora? \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and encouragement. It is also great to hear that you highlight our paper as well-written and easy to read. It is our goal to reach out to a wide range of readers who may not be native English speakers. \n\nWe kindly disagree with your comment that this is a paper suited for a journal. A position paper is different from a survey paper in the sense that we mainly focus on the papers that release code-switched data sets rather than all the papers published in the area of code-switching. Representativeness is a crucially important topic for code-switching research considering the variation across speakers and contexts it covers. We believe our paper will benefit the audience of this conference immensely to inform themselves about the issue since  they have much less chance to read about this topic with references across disciplines (e.g. linguistics, psycholinguistics, multilingualism) and an in-depth analysis of code-switched data sets (both speech and social media) published in computational linguistics venues. The lengthy references do not necessarily qualify a paper for a journal. Therefore, we kindly ask you to consider your argument on this matter again. \n\nTo address your question, we agree that our highlighted dimensions of variation may be applied to other forms of linguistic corpora. However, it is beyond the scope of this paper to cover all language related data sets in all areas. Instead, our goal is to focus on code-switching research specifically to make it manageable and in alignment with the “Multilinguality & Linguistic Diversity Track” of the current conference. To clarify your point, in section (5), we mention the guidelines for collecting and preparing representative code-switched data sets taking variation into consideration. Although there are some existing guidelines for data (e.g., in computational linguistics and machine learning), there are also issues specific to representativeness in data collection and preparation in multilingual and code-switched communication which we present as a check-list. For example, none of the earlier guidelines pay specific attention to location or register based variation or socio-demographic variation (not only for speakers but also for data collectors, transcribers and annotators) or data filtering favoring code-switching (i.e, opting only for code-switched data but ignoring the fact that the same speaker/user may also communicate in the monolingual mode)  which are crucially important for code-switching research. We will indicate this difference with an additional explanation in the discussion section more clearly and also highlight these unique aspects for code-switched data in the camera-ready version. Thanks for the suggestions, valuable insights and your time for reviewing our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The analysis takes into consideration virtually all corpora that represent phenomena of code-switching and does so in a non-anglocentric perspective and according to principles of true multilingualism.\nThe large amount of parameters considered by the authors for evaluating those corpora derive from linguistic and computational literature both.\nThe reflections provided by the authors may be rally useful to shed a novel linght and advance in this research area.", "reasons_to_reject": "The paper has mostly the form of a survey. Therefore publication in a journal seems, in my opinion, more adequate. In this case, more details can be presented and discussed, while, in the current version, only a limited amount of them can be presented.", "questions_to_authors": "The arguments provided for the evaluation of the analyzed corpora are interesting and allow the observation of several features and respects. \nNevertheless, in my opinion, most of them are not features specific to the code-switching corpora. For instance, the dimensions of variation discussed in the paper can be observed also in corpora where code-switching does not occur. Can the authors better discuss this point and organize the dimensions according to the fact that they refer or not only to code-switching corpora? \n\n", "paper_topic_and_main_contributions": "The paper presents an analysis of existing collections of texts in which code-switching occurs.\nIt introduces several parameters for the evaluation under several respects of the surveyed corpora, considerations about them and proposals for improvement in this area.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The authors provide a huge amount of references, probably too many for a conference paper. This raises in me the impression that the goal is to write a survey rather than another form of research. ", "Typos_Grammar_Style_and_Presentation_Improvements": "In my opinion (I'm not a native English speaker), the paper is well-written and organized and I've not detected typos. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper surveys a large number of codeswitching data sets regarding their representativeness and puts forth the position that progress in developing systems that correctly handle real-world codeswitching may be caused by failures to accurately represent the codeswitching communities and to clearly report information regarding representativeness (demographics, etc.) of the subjects and annotators.

One point of confusion in the reviewing process was what the author’s intentions were for this paper. While the paper does not explicitly identify itself as a position paper, it does explicitly state it is not aiming for a comprehensive survey [091-093]. However, the reviewers generally identified it as a survey paper, and I think most readers would come away with that impression. The authors clarified in the response period that they believe this is a position paper, and not a survey.

Why does this point matter? There appears to be some confusion about what the main contribution of the paper is, and it is not easy to determine from the paper itself, judging from the reviews.

As reviewer sZBB points out, if we take this paper as a position paper, the position is not well-supported by the paper itself:

> The main reason to reject this paper is that some of the main claims are not backed up with sufficient analysis and/or citations. In particular, I believe that the main claim of the paper is that a lack of representativeness in training and evaluation data leads to down-stream systems which are not fit for purpose (see lines 85, 694). However, there is no evidence in the paper which backs this up (empirical work here would be really good). 

As reviewer 65nB states, “empirical evidence for the main motivation is limited.” Reviewer 5PvD seems to believe this is a survey paper and that essentially no clear position is put forward.

If there is a central position, perhaps it is best summarized in the abstract:
> we argue that the lack of representative CSW data collection and preparation procedures could lead to this drawback.

However, as the reviewers point out, while this hypothesis is floated, there is no clear evidence given to support it. Significant issues of representation are raised, but the empirical evidence given mostly has to do with system adaptation issues. Systems adapted to improve on one population or data type show performance reductions in another, etc.

The structure of the paper could be improved to make it clearer what its goals are and focus more on their specifics (see reasons to reject from Reviewer sZBB for more detail).

Strong claims about the progress of codeswitching systems are made, for example in the abstract “there is still not much progress in building successful CSW systems”. This is a controversial claim that is only supported in the paper with the limited evidence mentioned above. The general question of adapting models to work well on different domains, genres, registers, types of input, etc., goes far beyond codeswitching, and it is not at all clear that creating and using more representative data will address this.

Another concern raised by reviewers is the limited contribution beyond existing work (see Reviewer sZBB in particular), given recent significant surveys.

Overall, all reviewers were in agreement that portions of this paper represented significant contributions, however, the paper’s structure makes it difficult to understand the paper’s primary contribution and distinguish it from recent work that is cited throughout.

Minor notes:

1. Reviewer 5PvD raises the question of whether a survey paper is appropriate for this venue; it is absolutely appropriate, as survey papers and position papers are explicitly called out as welcomed in the call for papers. I believe regardless of the exact framing of the paper, this content is absolutely welcome and encouraged at this venue.

2. While the reviewers did not point this out, I would like to make sure the authors are aware that the collection and reporting of demographic data about the speakers and annotators in corpora is often restricted by research ethics boards. For example, some Institutional Research Boards (IRBs) in the US do not allow this data to be recorded, and if it is recorded, they may not allow it to be released along with the data. In my personal experience with IRBs, as the annotators are not the subjects of research, I have never been able to convince an ethics board to allow the recording or release of this information.
Decision: Accept-Findings