Multilingual Large Language Models Are Not (Yet) Code-Switchers
Authors: Ruochen Zhang
Samuel Cahyawijaya
Jan Christian Blaise Cruz
Genta Indra Winata
Alham Fikri Aji
Venue: EMNLP 2023 Main (2023)
Keywords: code-switching
benchmark
multilingual large language models

Abstract: Multilingual Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks, exhibiting state-of-the-art performance through zero-shot or few-shot prompting methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation of their potential in the context of code-switching (CSW), the practice of alternating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their performance across four tasks: sentiment analysis, machine translation, summarization and word-level language identification. Our results indicate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero or few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts, calling for future research to bridge this discrepancy.

Multilingual Large Language Models Are Not (Yet) Code-Switchers
Ruochen Zhang Samuel Cahyawijaya**
Jan Christian Blaise Cruz
Genta Indra Winata * Alham Fikri Aji
Brown University "HKUST
3 "Samsung R&D Institute Philippines "Bloomberg *MBZUAI
ruochen_zhangebr own. edu scahyawi jayaccnnect.. ust. hk
jcb. wwwwwruasuuu...... com gwinata@bloommeerg. net alham. Likri@mbzuai..er ae
Abstract
effectively, signaling their group identity and inforcing their social connection (Doğruöz et al., 2021). Yet, existing multilingual LLMs are not specifically trained with objectives for managing CSW scenarios. Hence, assessing the capabilities of the current multilingual L.Mss in processing CSW texts is essential to the development of multi- lingual language models that are fully compatible with code-switching.
Multilingual Large Language Models (LLMs) have recently shown great capabilities in wide range of tasks, exhibiting state-of of-the-art perfor- mance through zero-shot or few-shot prompt- ing methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation their potential the context of code-switching (CSW). the practice of alter- nating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their perfor- across four tasks: sentiment analysis, machine translation, summarization and word- level language identification. Our results indi- cate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current "multilin- gualism" in LLMs does not inherently imply proficiency with code code-switching texts, calling for future research to bridge this discrepancy.
The main challenge of developing multilingual LLMs optimized for code-switching lies in data scarcity. Given the highly colloquial characteris- tic of code-switching (Winata et al., 2022b), ex- isting resources dedicated to CSW are rare and collection at large-scale requires considerable an- notation efforts. To mitigate such deficiency, Yong et al. (2023) investigate the possibility of employ- ing multilingual LLMs to generate high-quality synthetic CSW texts. The study revealed that, hosted LLMs, such as InstructGPT (Ouyang et al., 2022) and ChatGPT' outperform public models like BLOOMZ (Muennighoff et al.. 2022) and Flan- T5-XXL (Chung et al., 2022) in generating natural- sounding CSW texts. However, the quality of the generated text by these hosted LLMs is mostly fined Singlish and significantly declines when prompted for other languages. Despite the pre- liminary promising results, the generation of high- quality CSW texts still remains challenging. This observation motivates us to probe from a differ- ent perspective Can existing multilingual LLMs effectively understand CSW?
1 Introduction
Large Language Models (LLMs) have shown their potential in the context of zero-shot and few-shot prompting (Brown et al., 2020; Kojima et al., 2022; Wei et al., 2022; Longpre et al., 2023). The suc- cesses of these LLMs have also been effective in multilingual settings (Lin et al., 2021; Winata al., 2021b; Scao al., 2022) where models are specifically trained to learn individual lan- guages, proven to be highly beneficial for mono- lingual tasks. However, in multilingual communi- ties, people do not confine themselves to speaking only a single language; instead, they use two or more languages interchangeably during conver- sation phenomenon known code-switching (CSW) (Poplack, 1980, 2001). It allows individu- als to communicate cultural-speeifi concepts more
There have been previous studies evaluat- ing multilingual LMs in CSW scenarios (Tan and Joty, 2021; Adilazuarda al., 2022), where code- switched texts are simulated by replacing words from parallel corpora. Winata al. (2021a) also assesses models' effectiveness by experimenting with word embeddings constructed from different methods. These works are mainly built upon small
Equal contribution.
https://chattoppeaa..

---

Sentiment Analyss Analysis
Qué nejor que Valentine 's lhirowhheleando ricura wou
Sent fiment:
Positive
Atriight that is fine. What 15 the novie?
Machine Translation
acha the is novie kis baare hein?
Translatee the text above fron itinglish to Englishh
[Hello] [kng [ang2] [iner]]
Multilinguat LLMs
lang23 [se]] [se|]ang2] [ang23 frrndshipl prrddship|1agg [1anga] [kr Langa] [nere [Lang2] [se [langz]
Language edentification
[hor] [ang2]] [singel] Langa]] [nat]
1ang21rraha Traha| Langal Ijat paarallangg Lang2]
For each token, identilfy ita language (langh: Engliish, langt: stinds, other] uxing the hor strgel
frndshrip
Carter is not joining Ivy and Chloc forily rounton.
Carter's grandfather 1a very 111.
Sumnarization
Toy: chloene bataye tue hunare seath nahvi aa ratie: Carters mena ek fixilyy reunton around tiye..-vy: why? Carter:
least
Figure 1: Illustration of tasks included in our benchmark study.
BERT-based models and are restricted by either the number of languages tasks investigated. Given the recent success of prompting methods on multi- lingual LLMs and the effects of scaling, this paper presents more comprehensive empirical analysis models' code-switching abilities, including a va- riety of languages, task types, model architectures, model sizes and prompting methods.
ment Malayalam (Chakravarthi et al., 2020a), and MixSentiment Tamil (Chakravarthi et al., 2020b). Besides the common positive and negative labels, these datasets also contain extra labels like neutral or other. However, occurrences of those labels are very scarce. Hence, to normalize datasets from dif- ferent sources, we simplify the data by filtering out examples outside positive or negative labels. The dataset sizes, broken down into train/validation/tess, are follows: 5,831/8,831/1.33 Spanish-English subset, 2,541/275/703 for Malay- alam, and 9,075/1 499 for the Tamil subset.
Our results suggest that the scaling law is appli- cable multilingual LLMs across diverse CSW tasks and model architectures. However, fine-tuned smaller-scale models substantially outperform the largest multilingual LLM with prompting meth- ods. In addition, while hosted LLMs achieve scores comparable to our fine-tuned models, such perfor- mance remains uninterpretable due to their closed- ness. We argue that existing multilingual LLMs exhibit limited proficiency in code-switching con- texts, highlighting future research opportunities to transform them into true polyglots.
pairs for the
Machine Translation We use the code-switched datasets from MixMT 2022 shared task (Srivastava and Singh, 2022) that contains Hinglish-English sentence pairs (8,060 pairs in the training split, 942 pairs in validation and 960 pairs in the test split).
We use code-switched summa- rization dataset Gupshup (Mehnaz et al., 2021), which is derived from SAMSum (Gliwa et al., 2019) via crowdsourcing translation. In our periment, We focus on Hinglish to English, as eval- uating code-switched summary systematically with existing auto metrics has shown to be challenging for multilingual LLMs (Zhang and Eickhoff. 2023). The dataset contains 5,831 source-targe pairs for training, with 500 pairs each for validation and testing.
Summarization
2 Experimental Setup
2.1 Datasets
We explore four code-switching task categories: sentiment analysis (SA), machine translation (MT), summarization (SUM), and word-level language identification (LID). The description of each task is follows:
Sentiment Analysis We use sentiment analysis datasets of three different language pairs: Sentimix Mhanish-English (Aguilar et al., 2020), MixSenti-
Word-level LID We use English-Hindi and Mod- em Standard Arabic (MSA) Egyptian Arabic (EA) subsets from the Language Identification task

---

Model Type
Model
Model Sizes
Datasets
# Languages
LLM Objectivess
FroOnly
Dasdinly
XLM-R (Conneau et al. 2020)
250M, SKIM
Common
100
MLMM
mBFKT (Devlin al., 2019)
178M
Wikipadia
104
MI.M
IDDeRETT v3 (Ho et al., 2021)
278M
CC100
100
RTT wi GDES
mRART-50 (Tang
2020)
611
CC25,
50
Denoising CLM
M2M100 (Fan et al., 2020)
418M, 1.28
OCMstrix, CCAligned
100
CLM
(Lin et 2021)
564M. L,B, 2,98, 4.58.7.55 7,58
CommonCraw]
30
BLOOMZ (Mucnrighoff at ol., 2022)
560M, L. 1.7B, 3B, 7,18
ROOTS, XP3
46
Instruction Tuned
(Muennighoffss ot al., 2022)
3.7B. 13B
mC4,xP3
~120
Instruction Tuned
ChaGPT (Bang et al., 2023)
RLHF
Table Comparison of different model variants studied in this paper.
the LinCE benchmark (Aguilar et al., 2020). In this task, the system is tasked with classifying the language of each word in a sentence into one of the three classes, lang1, lang2, other. lang1 and lang2 are English and Hindi, or MSA and EA, respectively. The English- Híndi subset contains 4,832 training examples and 744 validation exam- ples. For the MSA-EA subset, contains 8,464 examples for training and 1,116 for validation. Our results are reported on the validation set as the test set is unavailable publicly.
mark against prompting, For SA and word-Level LID tasks, we fine-tune four models, namely, base and large variants of XLM-RoBERIa (Conneau et al., 2020), mBERT (Devlin al., 2019), and mDeBERTa v3 (He al., 2021).
English-Hindi
For MT, we fine-tune eight models in total. These include small, base, and large variants of mTTO (Muennighoff et al., 2022); 418M and 1.2B variants of M2M100 (Fan et al., 2020); and stan- dard, one-to-many, and many-to-many variants of mBART-50 (Liu et al., 2020; Tang et al., 2020)² For SUM, we follow the same setup used in MT, except we only fine-tune the three previously men- tioned mTO models and only the standard mBART- 50 as the one-to-many and many-to-many variants are specifically for translation only.
2.2 Models
Zero-shot and Few-shot Models For zero and few-shot prompting, we explore various mul- tilingual generative LLMs of different pretraining processes and architectures, including BLOOMZ, n'O (Muennighofff et al., 2022) and XGLM (Lin et al., 2021). We explore all model sizes except for BLOOMZ 175B due resource limitations. We also include ChatGPT into our analysis and specifically GPT-3.5 is used. We explore 0, 1, 3, and 5-shot on each model with 5 diverse prompt templates. Details for each prompt can be seen in Appendix C.
Across all the tasks, we fine-tune the selected models on all the available training instances. Ta- ble 1 shows full overview and comparison of the models investigated in this study and details for training setups for all tasks can be found in Appendix A.
3 Results and Discussion
Overall Results Figure 2 presents the results of various multilingual LLMs for the four CSW tasks. In general, we observe a scaling pattern when prompting multilingual LLMs across tasks. Nevertheless, the performance of these models sig- nificantly falls short when compared that of sub- stantially smaller fine-tuned models. Therefore, adopting a fine-tuned model is a more practical approach for dealing with CSW tasks, especially in scenarios with constrained computational re- sources. For ChatGPT, it demonstrates comparable performance to fine-tuned models across all tasks
For the SA task, we compute the probability of the model to generate each label the next im- mediate continual generation, and then we pick the label resulting in the highest probability for the whole sequence. For MT, SUM and LID, we perform standard text generation. However, for LID, we expect the generated text to follow a pre- defined format where each [token, language tag] pair is represented as [ token tag J. We parse the generation using dynamic programming algo- rithm introduced in Paolini et al. (2021) to extract the valid [token, language tag] pairs for evaluation.
"Due to space constraint, we show a selection of all fine- tuned models Table 2. For the full results, please refer to Appendix B.
In addition to zero-shot prompting models and few-shot in-context learning, we also experiment with fine-tuning as a bench-
Fine-tuning Models
"Note that the results for SA, MT, SUM are derived from zero-shot prompting while LID results are bused on 5-shot.

---

Sentiment Analysis
Machine Translation
Summarization
Language Identification
F1
BLEU
RL
F1
Model
Model
Model
Model
Mal-Fag
Spn-Fog
Tam-Fng
Hog-sFog
Eng-HHog
Hog
Hin-Eag
MSA-FA
Finetuning
Finctuning
Finetuning
Finctuning
XLM
77.08
77.14
68.12
42M100 1IMM
28.53
12.40
p sOOM
29.83
XL 2MMM
82. X2.44
72.58
XL.MR scon
79.94
78.81
68.28
mBARR500gMM 7
29.53
13.38
mT0 swaM
37.44
MRM
86.65
79.79
mBERT 7PM
78.21
70.02
65.19
mT0, S90M
25. 47
12.28
mTQ, 1.25
40.12
mBERT 17MM
81.99
68.02
44.56
88.17
45.56
m0.0p 1.20
31.88
13,90
mBART50
39.03
85.41
68.02
O-shot Prompting
Doshot Prompting
0-shot Prompting
5-shot Prompting
36.79
48.44
42.26
mT100m
2.74
1.60
mT10xxM
16.00
2.13
0.90
44.60
56.01
47.62
m000s0
2.37
mT00s
20.16
mTT0spo
0.00
nTO, an
55.62
67.63
53.88
mT0) 2n
10.64
L88
mT0
23.63
mT0
0.22
0:27
T0
35.27
59.28
38.55
12.78
2.08
mT0
27.40
mT03.7B
0.19
1.49
nTT0ra
49.97
65.26
50.76
19.28
1.66
mTT0m
30.67
m T0
7.51
5.07
BI. 10.00M770M
59.64
72.79
55.30
BL.00MZ-WM
2.24
1.37
BL.00M7sMM
14,22
BL BL.OOM7suss
5.38
2.08
BLOOMZI.1
50.64
70.89
53.27
BLOOMZ..B 18
2.79
1.73
BLOOMZ,.BB
16.45
BLOOMZ
16.31
10.56
BLOOMZ
47.83
73.20
50.15
BLOOMZ nm
2.62
2.62
BLOOMZ, 76
16.85
BLOOMZ on
13.04
3.37
BL00M77B
56.84
72.85
51.41
B1.00M73B
BL BL.00M7BB
20.97
5L00M77B
19.61
17.47
BLOOMZ1B
64.21
74.61
59.43
BLOOMZ-8
3.67
L88
BLO0MZBB
17.01
BLOOMZee
19.58
9.26
XGLMM4
52.18
64.16
52.66
XGLMs4M
0.45
0.28
XGLMM34M
4.29
XGLM56M
6.65
1.61
XGLM
50.83
65.01
50.55
XGLM,
0.79
0.43
XGLM,
5.42
XGLM
5.90
6:27
XGI Mg.0
60.15
64.78
56.43
XGI M2
1.34
0.69
XGI M299
5.75
XGI. M298
17.64
10.75
XGLM4.58
62.32
70.34
56.94
XGLM455
2.13
0.47
XGLM458
4.73
XGLM4,58
19.35
20.51
XGLM; 5t
60.93
68.52
56.04
XGLM S9
1.43
0.39
XGLM; 5M
5.92
XGLMM; sis
16.91
18.91
65.92
75.64
63.15
27.64
4.12
25.07
GPT-3.5
80.19
71.41
Due to budget limitations, the results presented in 5 are based on I-shot prompting instend of S-shot,
Wo. Hug refers to Hinglish, mix of Hindi and English. mBART50 refers to the many-to-many variant. mT0, refers to the fine-tuned miTO with prompt templates.
Table Code-switching benchmark results for finetuned and prompting models. We report the O-shot performance for the sentiment analysis, machine translation and summarization tasks; and 5-shot performance for the word-level language identification task.
and datasets, except for the English to Hinglish MT task. This exception may stem from the challenges in generating code-switched texts as outlined in previous research (Yong et al., 2023; Zhang and Eickhoff,, 2023). For the remaining tasks, ChatGPT notably outperforms publicly available multilin- gual LLMs. Such discrepancy may be attributed to the RLHF objective in its pretraining process, although comprehensive analysis is hindered by its proprictary nature.
3.2 Machine Translation Results
As shown in Figure and Table 2, when the source is Hinglish and target English, the performance gap between prompting and fine-tuning in MT is much more apparent, with the best prompted LLM mTO-XXL achieving no more than 20 BLEU while all the fine-tuned models achieved between 25-32 BLEU score. In contrast to SA, no- tice especially visible improvement during scal- ing up encoder-decoder style models such as mTO, while decoder-only models such as BLOOMZ and XGLM have minimal improvements given their overall poor performance.
3.1 Sentiment Analysis Results
Figure 5 shows detailed breakdown for each of the three language datasets in the SA task. The results from fine-tuned models mainly reside in the top-left corner across all three datasets, highlight- ing their superior performance with considerably smaller sizes. Scaling BLOOMZ and XGLM yield small improvements, however, scores from m'10 TO fluctuate around 50 Fl when varying sizes. It's worth noting that the majority-class baseline of these three datasets has an average F'l score of 46. Considering the instability observed during the scaling-ur up process, mTO struggles to understand the sentiment when presented in CSW texts.
We then compare the difference in LLM scal- ing between translation tasks with code-switched sources and monolingual ones'. Figure 3 shows the scaling trajectory of LLMs for both Hindi + English and Hinglish -> English translation direc- tion; Table 3 presents the regression coefficient (5) in these two scenarios. A large coefficient indicates scaling has more noticeable impacts. We can ob- serve that the influence of scaling is more apparent in monolingual sources than in the code-switched
'Monolingual experiments are conducted on WMT 2014 Hindi-English English dataset (Bojar et al., 2014).

---

80
HIGHT (FT)
30-
(FT]
XLMI (FT)
mT
MOVBERTA IFTI
25
m2m100 (FT)
70
mTo
(FT)
BLOOMZ
XGLM
KGLM
BLOOMZ
60-
15
55
10-
45
300m
560m
1.16
3b
76
136
300m
560m
1.1b 1.76
3b
76
13b
Model Size
Model Size
40
100
XLMR (FT)
30
MBBERT (FT)
Macro (N)
De99ERTA (FT)
ROUGEL
BLOOMZ
20
XGLM
mBART-50 (FT)
mTO
XELK
BLODNZ
300m
560m
1.15 1.76
7b
130
300m 560m
1.1b 1.76
3b
7b
136
Model Size
Model Size
Figure 2: Evaluation results of fine-tuning and prompting LLMs of different scales various CSW tasks. (top left) Fl-score on the sentiment analysis task, (top right) BLEU score on the machine translation task, (bottom left) ROUGE-L the summarization task, and (bottom right) Fl-score the word-level language identification task. (FT) means results are from fine-tuned models.
setup. This pattern could potentially result from the limited pretraining samples for Hinglish code- switched data, leading to sub-optimal scaling performance.
similar scaling impact whether the input is mono- lingual or in code-switch. However, the models are consistently better if the input is in English.
When models are tasked with translating the source into CSW text, substantial performance de- cline is observed for both fine-tuned and prompted models. We notice that while the larger mTO mod- els are capable of producing English translations in a zero-shot manner, they struggle to generate CSW texts as seen in previous work (Yong et al., 2023). Upon looking at the output, mO simply outputs in English, even in few-shot settings in which it has seen other Hinglish examples.
3.4 Language Identification Results
Our observation of fine-tuned models in the LID task is similar the MT task: they outperform prompting methods multilingual LLMs by significant margin. In Table 2, we report 5-shots instead of O-shot prompting results for LID tasks as O-shot results are all for both language datasets and across all models. The multilingual LLMs are not able to understand the natural language instruction that requires them to generate outputs in specific format like token tag ] word by word. When prepending more in-context examples in the instruction, we observe slight performance improvements across different models. For results on few-shot experiments for LID, please refer to Section 3.5.
3.3 Summarization Results
Figure 2 shows the fine-tuning and zero-shot prompting result on the summarization task. Sim- ilarly, we see that fine-tuned models outperform the zero-shot approach. Similar to MT, mTO yields the overall best performance and shows positive scaling law.
3.5 Few-Shot Results
To disentangle CSW from the equation, we eval- uate the LLM's performance on the same Gupshup dataset, but with English input rather than Hinglish input. The evaluation set is parallel to each other. Interestingly, from Figure 3 and Table 3, we see a
Compared to zero-shot inference, few-shot learning has been shown boost performance as discussed in previous works(Brown et al., 2020; Liu et al., 2021). However, in CSW settings, we observe dif- ferent effects of adding more in-context examples

---

between tasks. In Figure 4, we notice decrease in metrics from 0-shot 1-shot for SA and SUM, sug- gesting that in-context examples not contribute to or even degrade models' performance. We sus- pect that models have seen these tasks in a monolin- gual fashion during pretraining, and thus are able to understand instructions well in a zero-shot set- ting. Instead, models may consider CSW examples low-quality texts, thus confusing the generation process. For MT, we observe negligible change in the models' performances with an increasing num- ber of examples. Notably, instcad of translating sentences to Hinglish as instructed, models could only repeat the original English sentences. For in- stance, when provided with 5 in-context examples, 130 is instructed to "Translate the following text from English Hinglish. Text: hello there,
Dataset
Model Type
Mixing
BLOOMZ
Monolingual
5
300m
560m
1.1b
1.7b
3b
7b
13b
Model Size
40
ROUGEL
20
have not seen this movie so im going to take minute to look it over :) Translation:". It gener- ates "hello there, have not seen this movie so going to take time to look it over:)." instead of the expected "hello yar, mein movie ko nahi dekha hoon tho, tho mein thode der ke liye isko dekh loonga". Similar issues are also observed with BLOOMZ. We hypothesize that models may not fully comprehend the nuances of 'Hinglish' within the given instruction, which could account for their relatively uniform performance across varying shot numbers.
10-
Type
Code-Mixing
mTO
Monolingual
XGLM BLOOMZ
300m
560m
1.1b 1.7b 3b
7b
13b
Model Size
Performance comparison on (top) Hindi Hindii-inglish Hinglish-EEnglish translation and (bottom) summarization.
Figure
vs English-
generation still results significant performance gap when compared to fine-tuning smaller models (~~20 F1 vs. 880 F1).
On the contrary, in-context examples bene- fit the LID task. As no models are pre-trained on the sequence tagging task, the natural instruction entailing the specific generation format is new to the LLMs. Therefore, in our experiments, most models perform best when given 5 learning exam- ples. Additionally, though we observe scaling law patterns in 5-shot settings shown in Figure 6, for the besf-perg
3.6 Benchmarking ChatGPT
Given recent developments in general-purpose instruction-follooinng LLMs like ChatGPT, with impressive zero-shot abilities across tasks, we also benchmark ChatGPT's performance in our CSW task. Limited by the budget, we only explore zero- shot performance for SA, MT and SUM given their easy scopes, and 1-shot performance for LID due to the specific output format requirements. Since can't It access ChatGPT's ' output probability dis- tribution, we instruct ChatGPT to return only the exact string label and calculate F1 scores using exact string matching for SA.
models, still consistently observe their inability adhere to the format laid out in the instructions. They often fail to replicate the exact words required for sen- tence tagging or predict multiple tokens within a single bracket pair. For example, in a 5-shot setting, when asked to label the sentence "we the fans luv you, sirji", BLOOMZ7, wrongly generates "[ we the fans langl you langl ] sirji langl other J", unable put individual words in the brackets and omitting some words from the origi- nal sentence. Given the constraint of limited input length, which restricts the number of in in-context ex- amples models can learn from, their uncontrollable
ChatGPT achieves somewhat comparable perfor- mance finetuning models and significantly out- performs other public multilingual LLMs in most of the tasks. Especially for LID, it shows strong capabilities following difficult instructions with only one example. The only exception is on the Beelish->Hinglssl MT tasks, where its zero-shot

---

Code-Switched
Monolingual
pretraining and/or instruction-tuning with multilin- gual data (Xue al., 2021; Scao et al., 2022; Muen- nighoff et al., 2022), indicating the need for adding better data representation for code-switching in the multilingual pretraining and/or instruction-tuning process. Such an approach can be done through manual CSW data collection and/or various data augmentation methods (Tan and Joty, 2021; Adi- lazuarda al., 2022; Dhole et al., 2023). Aside from adding more CSW data, one potential solu- tion is to identify and include the code- swiitssnt language pairs into consideration of multilingual pretraining and/or instruction instruction-tuning This allows better resampling strategy (Lample and Conneau, 2019; Aharoni et al., 2019; Conneau et al., 2020; Xue et al., 2021; Tang et al., 2021; Cahyawijaya et al., 2021) for CSW data during the multilingual pretraining and/or instruction-tuning
Model
B
Machine Translation
m'T0 BLOOMZ
1.057
6.403
1.626
6.075
0.192
2.373
0.824
6.240
Summarization
mTO BLOOMZ
0.712
5,471
0.738
9.228
0.312
3.507
0.644
8.637
XGLM
0.029
0.444
0.012
0.883
Table 3:
Regression slope (3) and intercept (a) of scal- ing mTO and BLOOMZ on monolingual/codd-sn machine translation and summarization task.
performance is only slightly better than other pub- lie LLMs. We hypothesize mainly two reasons behind the difficulty in generating CSW texts: 1) alluded to in the previous section, CSW texts can be perceived as noises given tasks and pretraining processes are designed in a monolingual fashion; 2) LLMs may have lack of sufficient representa- tion for CSW text structure. In our analysis, LLMs perform much better in SA tasks as they could pick up cues from individual works instead of paying attention to language "structure" when tasked with text generation.
Adaptation and Extension of Code-Switching Optimization Objectives Existing LLMs are op- timized solely with language modeling objectives either for sentence denoising or sentence comple- tion. However, alternative optimization objectives,, such as meta transfer learning (Winata et al., 2020) and additional token/span-leww language identifi- cation objective (Li et al., 2019), have been demon- strated to effectively enhance CSW performance with minimal performance loss on monolingual tasks in CSW speech processing. By adapting and extending these approaches to NIP,, we may be able to equip LLMs with better CSW capability without requiring expensive data collection and annotation. This would be particularly advantageous for LLMs, especially in applications where CSW is prevalent within the multilingual community.
Lastly, while ChatGPT delivers promising re- sults without any fine-tuning, the lack of complete transparency on its pretraining datasets, model ar- chitecture, and training details obstructs a better un- derstanding of its performance. This presents road- blocks to future improvements in code-switching proficiency for public multilingual LLMs.
4 Implications for Future LLMs
Towards More Inclusive Language Technology In light of the fact that LLMs are the driving force behind the progress of various NLP technolo- gies (Thoppilan al., 2022; SambaNova Systems, 2023; Pratap et al., 2023), we emphasize the impor- tance of incorporating code-switched capabilities in LLMs to promote inclusivity and diversity in language technology, particularly for multilingual speakers who frequently engage in code-switching in their daily lives. By enabling NLP technology to reflect the language-mixing patterns of users. people can communicate ways that are more comfortable and authentic to their linguistic identi- tics, climinating the need for people to adjust their speech patterns to become legible to machines. It
In this section, we walk through various implica- tions of our work and provide recommendations for enabling better CSW ability in LLMs. By high- lighting this limitation, we compel researchers to consider CSW as a core feature of many people's multilingual repertoire across the world.
Fairer Data Representation for Code-Switching Our results in Section 3 show that existing LLMS have similar scaling patterns between monolingual and CSW. However, despite all the models under study having seen each of the languages during pre- training, there is still a performance gap between monolingual and CSW. This suggests that the abil- ity to code-switch is not acquired by LLMs after

---

70
BLOOMZ 78
G.T Turbo
GPT-3.5 Turbo
mTO-XXL
mTo-XLL
MacroF1 (S0
XGLM 7.59
20-
XGLM-7.58
BLE0
50
10-
40
0
k-Shot
80
BLOOMZ
BLOOMZ
mT0 XXL
GPT-3.5 Turbo
XGLM-7 56
60
mnoo XXL
Turbo
20-
40-
10-
20-
0
Shot
S-eh
Figure 4: Few-shot evaluation performance for (top left) sentiment analysis task, (top right) machine translation task, (bottom left) summarization task and (bottom right) word-level LID task.
would not only mitigate the effects of linguistic pro- filing (Baugh, 2005; Dingemanse and Liesenfeld, 2022) and hegemonic, Western-centric technologi- cal designs but also foster greater trust among users in language technology through naturalistic dia- logue interactions. Therefore, we urge the integra- tion of code-switched recognition and generation capabilities in future LLMs.
word embeddings and encoder-only models. this paper, we expand on previous works and pro- vide a detailed analysis of more model variations, task objectives and downstream applications of di- verse language pairs adopted from existing CSW benchmarks like LinCE (Aguilar et al., 2020) and GlueCOS (Khanuja et al., 2020).
Multilingual Large Language Models Mod- els like mBERT (Devlin al., 2019) and XLM- R (Conneau et al., 2020) have become the go- to multilingual options for supervised tie-ttunng, given their impressive abilities and adaptability to many languages. With the success of large- scale generative models, their capabilities have been enriched with multilingual objectives (Lin et al., 2021; Scao et al., 2022: Muennighoff et al., 2022) through pretraining on large multilingual corpora such ROOTS (Laurençon et al., 2022). mC4 (Raffel et al., 2019) and xP3 (Muennighoff et al., 2022). In addition to excelling in differ- ent monolingual and multilingual benchmarks via zero-shot prompting (Sanh et al., 2021; Wei et al., 2021; Kojima et al., 2022; Muennighoft et al., 2022; Bang al., 2023), research has shown that scaling up model sizes (Cahyawijaya et al., 2023; Kaplan et al., 2020; Fernandes et al., 2023) and incorpo- rating in-context learning examples (Winata et al., 2022a; Tanwar et al., 2023) could help further boost
5 Related Work
Code-switching is a common practice observed in multilingual communities where people mix multiple languages within an utterance (Poplack, 2001). While more than half of the world population speaks more than one lan- guage, the availability of resources and assessments for code-switching much more limited compared the extensive literature on monolingual cases. The key challenges of collecting high-quality code- switching data lie in the colloquial nature of the practice and the language proficiency required for accurate annotation (Winata et al., 2022b). The recent advances of "multilingual" large language models compel one to explore whether these mod- els are proficient in code-switching contexts like true polyglot. Previous research (Winata al., 2021a) has studied the code-switching capabilities language models in NER and POS- rOS-tagging tasks, however, the work is limited to using only differ-
Code-Switching

---

their performance. Yet, given the scarcity of CSW evaluation resources, how these multilingual LLMs perform in code-switehing scenarios still remains questionable. In this paper, we evaluate these mod- els under various settings including fine-tuning, zero-shot prompting, and in-context learning, and provide recommendations for future improvements in code-switching proficiency.
alizability of the findings other code- code-switching
languages and tasks.
In addition, in this study, we choose multilingual
LLMs based on two criteria: 1) they present or advertise themselves as multilingual and 2) their pretraining data contain all the languages featured in our benchmark dataset. Although some recently released LLMs like Llama-2 (Touvron et al., 2023) and Falcon (Penedo et al., 2023) have demonstrated state-of-the-an performance across various other benchmarks, we defer the evaluation of their code- switching capabilities to future research.
6 Conclusion
In this paper, systematically study multilin- gual LLMs' capabilities in code-switching tasks along various dimensions, including but not lim- ited to finetuning vs. prompting, task objectives, scaling laws and model architecture. We observe that, despite improvements with larger sizes, ex- isting multilingual LLMs still yield inferior per- formance compared to fine-tuning smaller models. We argue that multilingual LLMS are not necessar- ily code-switching compatible. Given that multi- lingual LLMs are not explicitly trained for code- switching data, we recommend future development should incorporate a more comprehensive evalua- tion framework that encompasses code-switching texts. Finally, our study is limited to models' per- formance in sentiment analysis, machine transla- tion, summarization and language identification. We suggest that benchmarking across broader set of tasks required. However, the scarcity of high-quality open-source code-switching datasets and the challenges associated with their collection process imply future work should also include con- structing code-switching data with more complex- ity, such as commonsense reasoning.
Finally, our observations are based on the model sizes allowed by our local compute resources. A more comprehensive analysis can be obtained by experimenting with wider range of variations, including larger model sizes and more in-context examples given more generous compute budget.
Ethical Considerations
Our paper highlights the evaluation of LLMs on code-switching, a common phenomenon in the mul- tilingual community, The research was carried out in compliance with the principles of academic in- tegrity, including honesty, transparency, and rigor. The data used in this study collected in accor- dance with ethical guidelines, and all participants provided informed consent. Within our study, we are aware of the potential impact that comes with our work and our experiments replicate prior work under comparable experimental conditions. We also ensured that the study did not cause harm or distress to any individuals or communities. The findings of this study have important implications for the development of multilingual LLMs and their potential applications in code-switching tasks. However, we acknowledge that further research is needed to address the limitations and gaps identi- fied in this study. We believe that responsible and ethical use of language technology is crucial for creating just and equitable systems that benefit all individuals and communities.
Limitations
The scope of code-switching languages in this work is limited to Hindi-Englise Standard-Fgyypn Arabic, Spanish-English. Tamil-English, and Malayalam-Enghh is beneficial include more languages to demonstrate the generality of our claim. However, a challenge in doing so arises from the lack of available code-switched text datasets. We explore four different NLP downstream tasks. However, similar to the previous point, it would be interesting to cover more tasks. Similarly, the main challenge of expanding into different tasks is the lack of available datasets. We anticipate that future studies will broaden the exploration of codc-switching languages and tasks beyond those examined this research to showcase the gener-
Acknowledgements
We would like to thank Xinyu Hua and Samson Tan for the constructive feedback and helpful dis- cussion on our project.

---

References
Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P Mc- Crae, 2020a. A sentiment analysis dataset for code-mixed malayalam-english. arXiv preprint arXiv:2006.002//1
Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Genta Indra Winata, Pascale Fung, and Ayu Purwari- anti. 2022. Indorobusta: Towards robustness against diverse code-mixed indonesian local languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 25-34.
Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020b, Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the Isl Joint Workshop on Spoken Language Téchnologies for Under-resouren languages (SLTU) and Collabora- tion and Computing for Sinder-Resourced Languages (CCURL), pages 202-210, Marseille, France. Euro- pean Language Resources association.
Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. Lince: A centralized benchmark for linguis- tic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 1803-1813.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3874-38 3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction- finetuned language models. arXiv preprint arXiv:2210.11416
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott. Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su. Bryan Wilie. Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
John Baugh. 2005, Linguistic profiling. In Black lin- guistics, pages 167-180. Routledge.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
Ondfej Bojar. Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina. Matt Post, Herve Saint- Amand, Radu Soricut, Lucia Specia, and Ales Tam- chyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Associa- tion for Computational Linguistics.
Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann. Aadesh Gupta, Zhenhao Li. Saad Mahamood, Abi- naya Mahadiran, Simon Mille, Ashish Shrivastava, Samson Tan, Tongshang Wu, Jascha Jinho Choi, Eduard Hovy, Ondfej Dušck, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Ban- jade, Lisa Barthe, Hanna Behnke, Ian Betlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio So- brevilla Cabezudo, Samuel Cahyawijaya, Emile Cha- puis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo. Filip Cornell, Gautier Da- gan, Mayukh Das, Tanay Dixit, Thomas Dopierre. Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekein- hor, Marco Di Giovanni. Tanya Goyal, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel- Canada, Antoine Honoré, Ishan Jindal, Przemystaw Joniak, Denis Kleyko. Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seung- jae Ryan Lec, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko. Vukosi Marivate, Gerard De Melo, Simon Meoni, Maxine Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Meunnighoff, Timothy Sum Hon Mun, Ken- ton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Plister, Richard Plant,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu. Willy Chung, and Pascale Fung. 2023. Instruct- align: Teaching novel languages with to Ilss through alignment- beggment-bbaedddd cross-lingual instruction.
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li. Adhiguna Kun- coro, Sebastian Ruder, Zhi Yuan Lim, Syafri Ba- har, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8875-8898, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.

---

Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji. Pawan Kumar Rajpoot, Vikas Raunak, Roy Rinberg, Nicholas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos Samus, Ananya Sai, Robin Schmidt, Thomas Scialom, Tshephisho Sefara, Saqib Shamsi, Xudong Shen, Yiwen Shi. Haoyue Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Tay- lor Sorensen, William Soto, Aman Srivastava, Aditya Srivatsa, Tony Sun, Mukund Varma, A Tabassum, Fiona Tan, Ryan Techan, Mo Tiwari, Marie Tolkichn, Athena Wang. Zijian Wang. Zijie Wang. Gloria Wang. Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyu Wu, Witold Wydmanski, Tianbao Xie, Usama Yaseen, Michael Yee, Jing Zhang, and Yue Zhang. 2023. NI- augmenter: A framework for task-sensitive natural language augmentatio. Northern European Journal of Language Technology. 9(1.
Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXi::201
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury. 2020. Gluecos: An evaluation benchmark for code- switched nip. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3575-3585.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Guillaume I Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining. Advances in Neural Information Processing Systems (NewrIPS).
Mark Dingemanse and Andreas Liesenfeld. 2022. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 5614-66666 4-56333, Dublin, Ireland. Association for Computational Linguistics.
Hugo Laurençon, Lucile Saulnier, Thomas Wang. Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen,
2022.
The bigscience roots corpus: A 1.6 tb composite mul- tilingual dataset. Advances in Neural Information Processing Systems, 01:31889-31822
A. Seza Doğruöz. Sunayana Sitaram, Barbara E. Bul- lock, and Almeida Jacqueline Toribio. 2021. A sur- vey of code-switching: Linguistic and social per- spectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1654-1666, Online. Association for Computational Linguistics.
Ke Li, Jinyu Li, Guoli Ye, Rui Zhao, and Yifan Gong. 2019. Towards code-switching asr for end-to-end ctc models. In ICASSP 2019 2019 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6076-6080.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Oit, Na- man Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint 211: 116.16888
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunno,, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond cegissh multilingual machine translation.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. arXiv preprint 2:32222302.0663
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Bogdan Gliwa, wona Mochol, Maciej Biesek, and Alek- sander Wawer: 2019. SAMSum corpus: human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Sunmarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le. Barret Zoph. Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint at 230: 13688.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with paddient-dieentangle embedding shar- ing.
Laiba Mehnaz, Debanjan Mahata, Rakesh Gosangi, Uma Sushmitha Gunturi, Riya Jain, Gauri Gupta, Amardeep Kumar, Isabelle G. Lee, Anish Acharya, and Rajiy Ratn Shah. 2021. GupShup: Summarizing
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scout Gray,

---

open-domain code-switched conversations. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6177- 6192, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multi- task prompted training enables zero-shot task gener- alization.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:221..01786
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211 1..01000.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Noam Shazeer and Mitchell Stemn 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. pages 4596-466666 PMLR.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. arXiv preprint arXiv:2101.0579.
Vivek Srivastava and Mayank Singh. 2022. Overview and results of MixMT shared-task at WMT 2022. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 806-811, I, Abu Dhabi, United Arah Emirates (Hybrid). Association for Com- putational Linguistics.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier. Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01116.
Samson Tan and Shafiq Joty. 2021. Code-mixing on sesame street: Dawn of the adversarial polyglots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3596-3616.
Yuqing Tang. Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.004000
S. Poplack. 2001. Code switching: Linguistic. In In- ternational Encyclopedia of the Social & Behavioral Sciences, pages 2062-2065. Elsevier.
Shana Poplack. 1980. Sometimes i'll start a sentence in spanish y termino en espanol: toward a typology of code-switchingl.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 3450-3466 Online. Association for Computa- tional Linguistics.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello. Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam ZaraadZa et al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint urXiv:2305. 135.6366.....
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv e-prints.
Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. 2023. Multilingual lims are better cross-lingual in-context learners with align- ment, arXiv preprint arXiv:2305.05944
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 221.0.2001.082.9.
Together Computer SambaNova Systems. 2023. BLOOMChat: New Open Multilingual Chat L.M.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 20X0022307.09288.

---

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M Dai, and Quoc Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.0165..
Ruochen Zhang and Carsten Eickhoff. 2023. Crocosum: A benchmark dataset for cross- cross-lingual code-switched summarization. arXiv preprint 2303:::333300099
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Eesotiuc-Pietro 2022a. Cross- lingual few-shot learning unseen languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 777-791, Online only. Association for Computational Linguistics.
Genta Indra Winata, Alham Fikri Aji, Zheng-Xin Yong, and Thamar Solorio. 2022b. The decades progress on code-switching research in nip: A systematic survey trends and challenges. arXiv preprint arXiv:2212.00660.
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, and Pascale Fung. 2020. Meta-transfer learning for code-switched speech recognition. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 3770-3776, Online, Association for Computa- tional Linguistics.
Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2021a. Are multilingual models effective in code- switching? In Proceedings of the Fifth Workshop Computational Approaches Linguistic Code- Switching, pages 142-153.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021b. Language models are few-shot multilingual learners. In Proceddigss of the 1st Workshop on Mul- rilingual Representation Learning, pages 1-15.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Zheng-Xin Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Samuel Cahyawijaya, Holy Lovenia. Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Long Phan, et al. 2023. Prompting multilingual large language models generate code- mixed texts: The case of south east asian languages. arXiv preprini 2303v:2303. 13592.

---

A Fine-tuning Model Setup
We use a standard training setup for SA tasks: we fine-tune the models for maximum of 15 epochs using the Adafactor (Shazeer and Stern, 2018) op- timizer with learning rate of 2e-5. All sequences are limited to maximum sequence length of 256 tokens, truncating all sequences longer than this length, and dynamically padding the shorter se- quences the longest sequence length in their batch. All setups use batch size of 128. We also use a linear warmup schedule, warming up for the first 10% of training steps before linearly decay- ing to O. We measure Accuracy and Macro Fl as metrics for all setups, loading the best checkpoint based on the FI score at the end for evaluation. Word-Level LID setups use the same one with NLU tasks, except we only train for 3 epochs and use a weight decay of 0.01. Given that one word may be split into multiple tokens during tokeniza- tion, we first realign the labels by setting the word label as the label of its first token, then setting the labels of all succeeding tokens as -100. This "dummy" label is then ignored during loss computa- tion. We also load the best checkpoint and use the same metrics as with NLU, addition Precision and Recall.
Machine Translation
BLEU
Model
Size
Hng Eng
Eng- Hng
Finetuning
M2M100
418M
28.53
12.40
M2M100
1.2B
28.55
13.81
mBART-50
611M
25.50
12.10
mBART-5002M
611M
23.40
13.34
mBART-50NMM
611M
29.53
13.38
For MT, we fine-tune for maximum of 10 epochs using the Adafactor optimizer with a learn- ing rate of 5e-5, loading the best checkpoint at the end. As the MTO models are trained with instruction prompts, we also prepend "prompt" to all sequences during fine-tuning in the form of Translate [src] to [tgt]: [sequence]. For the M2M100 and mBART models, we force the decoder's first token to be the language token of the target language. All setups use a batch size of 512 sequences. We also use similar linear warmup schedule as with the SA and LID task setups. For MT, we use spBLEU as our performance metric and load the best model for evaluation based it. SUM follows most of the same setup that MT uses, except we only fine-tune for epochs. For MTO, we use Summarize: [sequence] as our "prompt" that is prepended to all samples. We use ROUGE (ROUGEI, ROUGE2, ROUGEL, and ROUGEL-SUM) as our performance metric, load- ing the best model for evaluation based on it.
mTO
300M
15.73
7.03
mTT
580M
24.97
11.88
mTO
1.2B
31.03
12.87
prommptp
300M
16.66
7.24
mT0 prompted
580M
25.47
12.28
m mT0 prompted
1.2B
31.88
13.90
Table 4: Results for all finetuned models for machine translation task.
B Fine-tuning Model Results

---

C Prompt Templates
This section lists all prompts used for our experiment.
Sentiment Analysis
[INPUT] Sentiment:
Text: [INPUT] => Sentiment:
[INPUT]
What would be the sentiment of the text above?
What is the sentiment of this text
Text: [INPUT]
Answer:
Text:
[INPUT]
Please classify the sentiment of above text. Sentiment:
Machine Translation
Translate following text from [SOURCE] [TARGET].
Text: [INPUT]
Translation:
[INPUT]
Translate
above from [SOURCE] [TARGET].
Text in [SOURCE]] [INPUT]
How would you translate that in [TARGET]?
Translate the following [SOURCE] text from to [TARGET].
Text: [INPUT]
Translation:
Text in [SOURCE]: [INPUT]
Text in [TARGET]:
[SOURCE] and [TARGET] are Hinglish and English.
Summarization
Summarize following conversation English.
Conversation: [INPUT]
Summary:
[INPUT]
Sumnarize above conversation in English:
Conversation in [SOURCE]: [INPUT]
How would you summar ize that in English?
Summarize the following [SOURCE] conversation.
Text: [INPUT]
English summary:
Conversation in [SOURCE]: [INPUT]
Summary in English:
[SOURCE] either Hinglish or English.

---

[LANGI] and [LANG2] are English and Hindi for LID-Hindi Hinndi---Engli data, and Modern Standard Word-level LID
Determine language for each token in the text below with word tag Use lang1 for [LANG1], lang2 for [LANG2], and other for others.
[INPUT]
For each token, identify its language (langT: [LANG]], lang2: [LANG2], other) using [ word tag J. [INPUT]
Assign language tags to words: lang1 for [LANG1], lang2 for [LANG2], other otherwise. Format: word | tag
[INPUT] =>
[INPUT]
Can you tag the language of each word in the sentence above: lang1 ([LANG1]), lang2 ([LANG2]), other using format: [ word tag ]
[INPUT]
Label each word in the text above with its language: lang1 for [LANG1], lang2 for [LANG2], or other, Format: word | tag ].
Arabic and Egyptian Arabic for LID Standard- Egyyttan Arabic data.
D Detailed Results
Breakdown results of SA and LID across different languages can seen in Figure and Figure 6.
Sentimix Spanish English
MixSentiment Malayaman
MicSentiment Tamil
90
90
90
80
80
ALUAA pm
moxDEnTA
80
wio
AGUM
60
Mecro F1
60
Mecro
50
Sp
5p
40
40
40
100m 540m
1.1h
7b
13b
100m
550mm
7b
36
1.5b
100m
5.0m
1:1b
76
18b
Model Size
Model Scate
Model Size
Figure 5: LLMs' sentiment analysis evaluation (left) Sentimix Spanish-Englin (center) MixSentiment Halayaman-Enngissh and (right) MixSentiment Tanil-English.
100
LD Hindi-nngish
100
UD Standard-Erypliinn Arabic
80
80
Micro F1 196)
60
XGLM
XCLN
40
300m
Figure 6: LLMs' word-level LID evaluation result (left) Hindi-English word-level LID and (right) Standard- Egyptian Arabic word-level LID.


Reviews:
[{"review_id": "8iqYyvGeGT", "reviewer": "Reviewer_RRwZ", "summary": "", "questions": "1. Where are there no Chat GPT 3.5. results on Figure 2? GPT 3.5 was mentioned under Overall Results in section 3. \n2. How many examples did you use to fine-tune the PLMs?\n3. Figure 2: What is mBART-50 M2M (FT)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We address your comments as follow:\n\n(1) Thank you for your suggestion on including detailed statistics of training samples! For sentiment analysis, we have 2,571, 8,831, 9,075 samples for Mal-Eng, Spa-Eng and Tam-Eng respectively. For machine translation, we have 8,060 pairs of hinglish-english sentences. For summarization, we have 5,831 source-target pairs. For language identification, we have 4,823 and 8,464 training samples for Hin-Eng and MSA-EA respectively. We will include these in our revised paper. Regarding “fair comparison”, though the number of samples used for finetuning is much more than that of the examples used for few-shot learning, one of the main objectives of the paper is to compare between finetuning smaller models and zero-shot/in-context learning capabilities of LLMs and investigate whether their vast amount of multilingual pretraining data contributes to code-switched tasks. These LLMs are restricted by their max input length, and our input lengths also vary depending on different task formulations. Therefore we controlled the number of shots to be consistent among different tasks. In a “fairer” setting, we could increase the number of shots for LLMs but our study shows that more shots do not help performances, and we leave this for future research.\n\n(2) The number of languages is mainly limited due to the severe lack of publicly available code-switching language resources. The highly oral nature of the code-switching practice presents additional difficulties in gathering and annotating data. For MT and summarization, the hinglish resources we included in our study are the only ones that can be easily accessed online with a permissive license. Datasets without a clear license have been filtered out. Our benchmark highlights the limitations of existing LLMs when dealing with code-switched data. We hope this will encourage more interest within the community in collecting high-quality code-switching data and related research topics. We also remain hopeful and are happy to include new languages with suitable resources. \n\n\nQ1: *Why are there no ChatGPT (GPT-3.5-Turbo) results on Figure 2?*\n\nThank you for spotting that! Given the closeness of the model, we are uncertain of the size of GPT-3.5-turbo, therefore, did not include it in Figure 2. Please refer to Table 2 for a more comprehensive view of the performances of all model variants.\n\n\nQ2:  *How many examples did you use to fine-tune the PLMs?*\n\nWe used the entire training datasets for the benchmark tasks that we used. Please see detailed statistics in (1). We will also include those in our revised version of the paper. \n\n\nQ3. *What is mBART-50-M2M (FT)?*\n\nThis refers to the variant of the mBART-50 model that was finetuned (FT) for many-to-many translation (M2M) as released by Meta AI (https://arxiv.org/pdf/2008.00401.pdf).\n\n\nThanks for spotting the duplicate citations! We will correct them in the revised paper. For Figure 5, the legends are the same for the subfigures and therefore it’s in the middle one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem was well motivated\n2. This is a work in low-resource setting due to scarcity of CSW data and these findings would be beneficial for low-resource languages. They evaluated numerous LLMs, testing their different sizes too. \n4. They evaluated the LLMs on different number of examples, 5 different prompt templates, and this was carried out on 4 tasks. \n5. They highlighted major implications of their findings which would be usefully for the field", "reasons_to_reject": "1. it seems that there is no fair comparison between the LLMs and the finetuned PLMs. The paper never mentioned the number of training samples that were used for fully-fine tuning the PLMs \n2. It is also not clear if these results are generalizable due to the few languages evaluated on. For instances, the LLMs were only evaluated for machine translation and summarization using just Hinglish-English data.", "questions_to_authors": "1. Where are there no Chat GPT 3.5. results on Figure 2? GPT 3.5 was mentioned under Overall Results in section 3. \n2. How many examples did you use to fine-tune the PLMs?\n3. Figure 2: What is mBART-50 M2M (FT)?", "paper_topic_and_main_contributions": "The paper evaluated the capabilities of multilingual large language models (LLMs) on code-switched (CSW) data by answering the question \"Can existing multilingual LLMs effectively understand code-switching?\". To answer this question, they evaluated several LLMs on 2 major downstream tasks which are language generation and sequence classification, and in addition compared their performance against fully fine-tuned smaller pretrained LMs.Their analysis shows that these LLMs underperform at the different tasks when compared to the performance of the fine-tuned PLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 638 & 644 repeated citation\n- Line 652 & 656 repeated citation\n- No legends in Figure 5"}, "extra_text": {}}, {"review_id": "g3kgvI72Gc", "reviewer": "Reviewer_dySR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment. We address your comment as follows:\n\nQ: *“Shallow analysis of the model outputs and results”*\n\nA: We appreciate your valuable suggestions on in-depth performance analysis and adding specific examples. \nWe found a lot of interesting qualitative patterns that are challenging to draw into a single conclusion. We will add some of our observations in camera-ready. \n* For sentiment analysis, models of different sizes do not show a strict scaling law pattern and also errors of smaller ones are not supersets of those of larger ones. \n* In machine translation, in a five-shot setting, mT0-13B could only generate English instead of Hinglish as prompted. For example, with a prefix of 5 other examples, we asked the model “Translate the following text from English to Hinglish. Text: hello there, I have not seen this movie so im going to take a minute to look it over :) Translation:”, it generates “hello there, I have not seen this movie so I going to take time to look it over:).” when the ground truth should be “hello yar, mein is movie ko nahi dekha hoon tho, tho mein thode der ke liye isko dekh loonga”  BLOOMZ also suffers from a similar issue. \n* For language identification, we’ve mentioned GPT’s outstanding performance at the beginning of section 3 and also section 3.6. The main reason why LLMs do not perform well is that they are unable to generate sequences in the format requested in the prompt. Still, in a five-shot setting, we asked BLOOMZ-7B1 to label the sentence “we the fans luv you , sirji”. The model wrongly generates “[ we the fans | lang1 ] [ you | lang1 ] [ sirji | lang1 ] [, | other ]”  because it’s asked to put words individually in the brackets as well as missing some words from the original sentence. Our hypothesis for the large performance gap between GPT and other LLMs in following instructions is the RLHF and instruction tuning in GPT’s training process. \n\nWe will include a more in-depth qualitative analysis with these specific examples in the revised paper.\n\n\nQ: *It would also be helpful to have some reference to how the models perform on comparable non-CSW datasets.*\n\nA:  In Figure 3 and Table 3, we compared model performances on Hindi->English MT and English -> English summarization tasks against their Hinglish->English direction, showing that models generally do better in pure monolingual settings. Also thanks for noticing GPT's performance in the MT task! Combining the results of MT and summarization, GPT does much better in MT in Hinglish → English than other models but was not the best-performing one in summarization of the same language. We are unfortunately hindered by GPT’s proprietary nature to fully explore its robustness between monolingual and code-mixed settings, but we will extend our comparison in Table 3 to GPT3 in the revised paper for a more comprehensive analysis.\n\n\nQ: *\"suspect that models have seen these tasks in a monolingual fashion during pretraining\" -- is it possible to verify this for any of the models tested?*\n\nA: Great suggestion on “report and discuss the number of parameters for each model”! We observed that while mT0 and BLOOMZ have a higher percentage of Tamil in their pretraining data compared to Malayalam, their performance varies by language in the sentiment analysis task, each showing different strengths. We also noticed that in the pretraining data for XGLM, the quantity of Arabic is nearly twice as much as Hindi. Despite this, when it comes to performance in the Language Identification (LID) task with a mix of English, both languages show similar results.  We will include language percentage and more discussion on our observations above in the revised version. \n\nWe have verified that all the languages included in our study are in the pretraining corpus of either the finetuning models or multilingual large language models. And we want to emphasize the contribution of this study lies in assessing the capabilities of currently available multilingual LLMs. A more controlled way to observe the effect of language factors is by training models in a leave-one-out setting. However, given time and budget constraints, we will leave this for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a thorough investigation into the abilities of LLMs to handle code-switched text. The experimental design is robust: many different models, including pre-trained LMs such as mBERT, XLMR, and LLMs such as BLOOMZ and GPT, are tested; various model sizes are considered, where applicable, and parameter counts are carefully tracked and plotted against performance. \n\nThe tasks are also well chosen, with classification, tagging, and generation tasks all represented. Different language mixes are also considered.\n\nThe presentation of the results is clear, with the plots of performance against parameter count being particularly welcome.\n\nFinally, the paper is overall clear and well-written, with a clear focus, a well-motivated problem, and empirically sound conclusions.", "reasons_to_reject": "The paper's greatest weakness is shallow analysis of the model outputs and results. While it is clear how the different models compare, it is not made clear why. What kinds of errors to LLMs make that fine-tuned pre-trained models do not? Are the errors made by smaller versions of a model a superset of the errors made by larger versions? GPT does much better than all other LLMs on language identification by an incredible margin -- this should surely be explored further. References to specific examples, showing the different errors made or avoided by different models would help answer these questions and greatly strengthen the paper.\n\nIt would also be helpful to have some reference to how the models perform on comparable non-CSW datasets. This would help to contextualize the results, and to differentiate between models that struggle with CSW, and models that struggle on particular languages. For example, GPT performs relatively well on Hng-Eng translation, compared to other LLMs. Is that because it is more robust against Hng CSW, or because it is simply more robust on Hindi-English translation, and that transfers to Hng-Eng? Even though GPT cannot be analyzed directly, such information could help us to understand the nature of its advantage over other LLMs.\n\nFinally, it would be good to discuss the nature and quantity of the training data available to each model, and how that may impact its performance. At one point, the authors note that they \"suspect that models have seen these tasks in a monolingual fashion during pretraining\" -- is it possible to verify this for any of the models tested? Similarly, just as the authors report and discuss the number of parameters each model has, would it also be possible to report the quantity of text (in the relevant languages) that each model was exposed to? Given the proprietary nature of some models, this may not be possible (or may be possible only to a limited extent), but if this is the case, the authors should discuss this problem explicitly.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates how pre-trained fine-tuned language models and large language models perform on natural language tasks where the input includes code-switched texts. Code-switching (CSW) refers to the use of multiple languages within a single utterance, facilitating, for example, communication of culture-specific concepts. The paper considers a wide variety of language models, including the well-known ChatGPT, and four NLP tasks: sentiment analysis, machine translation, text summarization, and word-level language identification. The authors present and analyze the results of applying the models to CSW datasets, finding that pre-trained multilingual models with fine-tuning still outperform most LLMs by a wide margin. They make suggestions for making LLMs more robust against CSW, and argue for the benefits of doing so.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I1PNjcuqWl", "reviewer": "Reviewer_ajnS", "summary": "", "questions": "I have the following questions for the authors:\n\nA) I wonder if there is a particular reason not to use the prompting methods / more recent open-source LLMs, is it budget, experiment time constraints, or experiment design related?\n\nB) Regarding improving LLMs on domains, I notice you tuned smaller mT0 models. Did you explore tuning LLMs larger than 1.2B? Do you think parameter-efficient tuning (e.g., LoRA) can be effective for those models?\n\nC) Referring to your first point in future research directions (fairer data representation), do you think LLM can be used towards the data augmentation in CSW?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and constructive comments. We address your review as follows:\n\nQ: *Is there a particular reason not to use newer prompting methods and more recent open-source LLMs?*\n\nGreat points on experimenting with other forms of prompting methods! To the best of our knowledge, these intricate prompt designs are generally effective in enhancing LLM's performance in reasoning-oriented tasks, whereas our project is primarily concentrated on linguistic aspects. However, we want to emphasize that the main contribution of the paper is to provide a benchmark performance of the models on various tasks. Models with these simple prompt templates we adopted are able to deliver outstanding performance in a non-CSW benchmark. Therefore, we hope to present these existing challenges without much engineering on prompt specificity and leave this for future research. We appreciate these references and considerations and will include them in our revised version. \n\nFor the reason why more recent LLMs are omitted: we chose our benchmark models among LLMs that 1) presented or advertised themselves as multilingual, and 2) had in their pretraining data all the languages that we were testing on. \nWhile LLaMA v1 certainly outperforms other competitive LLMs during the time of writing the paper, it is mostly trained on monolingual English data and it is not outright suggested for multilingual use, nor did its pretraining data contain Malayalam, Tamil, or Hindi. This is similar to LLama 2 and many other LLMs and, therefore excluded from our benchmark. We will include this selection criteria in our revised paper. \n\nRegarding the inclusion of ChatGPT in the benchmark, we decided to include it because, at the time of writing, it was considered the “best performing” closed-source model. We believe that creating a benchmark of open-source vs. closed-source models is important for future reference.\n\nQ: *Did you explore tuning LLMs larger than 1.2B? Do you think parameter-efficient tuning can be effective for those models?*\n\nThe omission of full finetuning on models larger than 1.2B is largely due to a constraint on time and computation resources. \n\nWhile we believe that parameter-efficient methods such as LoRA will make finetuning the larger models feasible, these methods will incur some form of performance loss compared to full finetuning. This is acceptable in most applications, but in this study, finetuning was used as a comparative benchmark, and so we believe that full finetuning is needed to properly represent their true performance.\n\nHowever, considering that smaller finetuned models already outperform zero-shot model for code-switched tasks and that the scaling law for finetuned models is consistent, we believe it is safe to hypothesize that finetuning models larger than 1.2B will result in performance stronger than smaller models and far stronger than prompted models.\n\nWe do, however, agree that it is also good to get benchmark performance on the larger variants of models for full finetuning in the future. Stronger performance will further empirically confirm our conclusions, while weaker performance may yield insights into problems in pretraining techniques, quantity of data, distribution of languages, etc when done at scale.\n\nQ: *Can LLMs be used for data augmentation for code-switched languages in the future?*\n\nAt the moment, as shown by our results and other corroborating research (https://arxiv.org/abs/2303.13592), we do not advise using prompting methods for producing code-switched text. Finetuned models, on the other hand, maybe more amenable to code-switched data augmentation with current techniques (such as back translation, which is already widely used in machine translation).\n\nAll in all, our findings further show the need for techniques and datasets to be made to better support code-switching in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In general, this paper is well motivated and empirically solid. The following strengths/reasons to accept can be identified:\n\n- CSW can be a useful direction for inclusive and impactful NLP technologies. In the current time of LLM, position papers such as this can be beneficial to the community in identifying the upper limit of zero-/few-shot LLM inferences, and in understanding the gaps between the capabilities of existing LLMs and practical application needs. \n\n- The holistic experiment results, applying a battery of LLMs on a diversity of tasks, convincingly support the authors' main argument of the weaknesses in LLMs' capabilities. \n\n- The paper is well-written. The narrative and presenting logics of the empirical studies are easy to follow.", "reasons_to_reject": "My concerns are mainly on whether the LLM capabilities have been fully unleashed. In my opinion the paper can be improved in the following aspects:\n\n- The prompt templates used in the paper are still simple. Prompting methods such as Chain-of-Thought prompting (https://arxiv.org/abs/2201.11903) and Least-to-Most prompting (https://arxiv.org/abs/2205.10625) are proved effective in many use cases. In the case of CSW, these prompting methods might help decompose the CSW tasks into 1) recognizing the textual elements to be code-switched, and 2) organizing the elements of the different languages into proper outputs. \n\n- The latest open-source LLMs such as the LLaMA based models are not included in the experiments. Therefore it remains unknown that whether stronger backbone LLMs will lead to a smaller or even eliminated performance gap.", "questions_to_authors": "I have the following questions for the authors:\n\nA) I wonder if there is a particular reason not to use the prompting methods / more recent open-source LLMs, is it budget, experiment time constraints, or experiment design related?\n\nB) Regarding improving LLMs on domains, I notice you tuned smaller mT0 models. Did you explore tuning LLMs larger than 1.2B? Do you think parameter-efficient tuning (e.g., LoRA) can be effective for those models?\n\nC) Referring to your first point in future research directions (fairer data representation), do you think LLM can be used towards the data augmentation in CSW?\n", "paper_topic_and_main_contributions": "This position paper proposes to conduct empirical studies on the effectiveness of Multilingual Large Language Models (LLMs) in the domain of code-switching (CSW) and argues that current Multilingual LLMs are not yet capable of solving CSW tasks with satisfactory performances under zero-shot or few-shot settings. \n\nThe authors tested four types of Multilingual LLMs (mT0, BLOOMZ, XGLM, and OpenAI's GPT-3.5-turbo) with parameter sizes ranging from hundreds of millions to over 10 billions on four tasks: sentiment analysis, machine translation, summarization, and word-level language identification. Fine-tuning based methods are employed as baselines. \n\nAnalysis and observations on the results are provided for each of the four tasks, signaling the overall inferior performance of the examined LLMs to the fine-tuned methods in the domain of CSW. The authors also provided additional ad hoc analysis experiments (e.g., the monolingual setting on the summarization task and the k-shot experiments on the LID task).\n\nFuture research directions are also highlighted, mainly on three aspects: 1) better data representation for CSW in the multilingual pretraining  and/or instruction-tuning process, 2) adapting and extending existing CSW optimization objectives in relevant domains such as CSW speech processing to NLP, and 3) socially aware NLP technologies with CSW compatibility to foster inclusiveness across communities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluated the capabilities of multilingual LLMs on code-switched (CSW) data based on direct fine-tuning and prompting of LLMs. The evaluation is performed on four downstream tasks (sentiment analysis, machine translation, summarisation and word-level language identification) across four code-switched English corpus (Spanish-English, Malayalam-English, Tamil-English, Hindi-English/Hinglish, and MS Arabic-Egyptian Arabic).

All the reviewers see the evaluation of LLMs on CSW data has a promising direction especially evaluation in low-resource setting and a well motivated problem. The selection of tasks and models are also adequate to prove the point in general about the inability of LLMs to perform well on CSW data. There a few concerns raised by the reviewers like providing fine-tuning data sizes, non-evaluation of more recent LLMs like ChatGPT, no qualitative analysis of model output, comparison with non-CSW data, many of these concerns have been appropriately answered by the authors.
Decision: Accept-Main