GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Authors: Md Tawkat Islam Khondaker
Abdul Waheed
El Moatez Billah Nagoudi
Muhammad Abdul-Mageed
Venue: EMNLP 2023 Main (2023)
Keywords: Arabic NLP
Arabic Dialects
ChatGPT
GPT4
TL;DR: We conduct the first large-scale automated and human evaluation of ChatGPT and GPT4 on Arabic natural language understanding and generation.

Abstract: ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.

GPt: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Md Tawkat Islam Khondaker
Abdul 5
Moatez Billah Nagoudi*
Muhammad Abgul-Mageed-6
Deep Learning & Natural Language Processing Group. The University of British Columbia
Depaatment of Natural Language Processing & Department of Machine Learning, MBZUAI
(tawkatēcs moatez. moateez n muhammad. mageed)@ubc.com ca
Abstract
Tasks
ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English bench- marks. However, the model's 's efficacy across di- verse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal vari- eties. Our comprehensive study conducts large- scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further un- dertake meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Di- alectal Arabic (DA), unveiling the relative short- comings of both models in handling Arabic dialects compared to MSA. Although further explore and confirm the utility of employing GPT-4 as a po- tential alternative for human evaluation, our work adds growing body of research underscoring the limitations of ChatGPT.
Evaluation Models
Learning
N-shot
BLOOMZ
MARBERTV2
Tully linetuned (aaanna)
Figure 1: Experimental setup for our evaluation. We evaluate ChatGPT on 44 Arabic tasks
attention due to remarkablc abilitics.
The underlying model behind ChatGPT is be- lieved be pretrained on large datasets from mul- tiple languages, which could enable the model to work well in multilingual settings. There is, how- ever, much hype about ChatGPT's abilities. In fact, some observations around ChatGPT remain anecdo- tal, calling for rigorous evaluation of its abilities in different languages. While multiple investiga- tions of ChatGPT performance on English cumulating fast (Qin et al., 2023a; Gilardi et al., 2023; Laskar et al., 2023), evaluation on other lan- guages remains largely speculative. This lack methodic evaluation inhibits our understanding the capabilities and limitations of LLMs beyond English. This motivates our work for evaluating ChatGPT on the wide collection of languages and language varieties commonly referred to Arabic. Arabic has rich morphologies and syntactic struc- tures. It also has a vast native speaker population of over 450 million people, making investigations necessary for technological inclusion. Combined with this large population, the complexity of Arabic
1 Introduction
Large language models (LLMs) pretrained on next token prediction brought significant progress to NLP. These models can be finetuned to follow hu- man instructions, allowing users to steer model out- put (Wei et al., 2021; Wu et al., 2021; Chung et al., 2022b; Ouyang et al., 2022; Muennighoff et al., 2022a). C is the most prominent among these models and has recently received significant
II this paper, we use the term ChatGPT to refer to the gpt-3.5-turbo-0200 model.
"httss//oppnai.cmm//bnagahgg

---

requires models to have deep understanding of expansive sociolinguistic contexts. This also makes the evaluation on Arabic all the more important to our understanding of how large multilingual mod- els behave, with implications that go well beyond Arabic.
4. We further conduct quality assessments on the models' generations using both human and GPT-4 as evaluators, finding GPT-4 evaluation to notably align with human evaluation.
5. Through our empirical analyses, we find that ChatGPT significantly lags behind much smaller Arabic-focused finetuned models on almost all tasks. Our findings should motivate future work focused at improving LLM per- formance on Arabic languages and varieties.
Concretely, our objective is systematically evaluate ChatGPT on a wide range of Arabic nat- ural language understanding (NLU) and natural language generation (NLG) tasks, identifying exist- ing gaps and ultimately possibly guiding the devel- opment of more effective Arabic and multilingual applications. Through our comprehensive bench- marking on 11 diverse tasks comprising over 60 datasets, we observe that ChatGPT exhibits inferior performance compared to much smaller finetuned models (Section 6 and Section 7). When we con- sider the performance of ChatGPT and GPT-4 on country-level dialectal Arabic as compared to Mod- ern Standard Arabic (MSA), the modern-day vari- ety used in formal settings, we reveal even more inferior dialectal performance (Section 8). To the best of our knowledge, this is the first work to study the comparison of ChatGPT and GPT-4 Arabic dialectal variation.
Related Work
We provide a brief overview of the previous works that evaluate ChatGPT on NLP tasks here. We offer detailed walkthrough in Appendix A.
Performance on English. In machine transla- lion (MT), while ChatGPT equates to commercial tools like Google Translate (Jiao et al., 2023), it falls short in domain-specific translation and low-resource languages. However, strategies like pivot prompting (Jiao et al., 2023) and additional prompts (Peng et al., 2023; Gao et al., 2023) can enhance performance. In question-answwriieee answering (QA), while ChatGPT shows potential (Zheng et al., 2023; Shen et al., 2023; Omar et al., 2023), it falters complex open- domain questions and tasks requiring reasoning (Zheng al., 2023; Tan et al., 2023) and is vul- nerable adversarial examples and perturbations (Shen et al., 2023). For text classification, ChatGPT performs well in zero-shot and few-shot settings (Zhong et al., 2023b; Gilardi et al., 2023; Wang et al., 2023b), sometimes matching or surpassing fully supervised models.
information-in
The present study unambiguously demon- strates that, notwithstanding its noteworthy per- formance standardized English benchmark datasets (Zhong et al., 2023a), the efficacy of ChatGPT universally applicable solution ap- pears be unsubstantiated when considering lan- guages characterized by extensive linguistic vari- ations, particularly in the context of Arabic. Suc- cinctly put, it evident that considerable enhance- ments can still be made pertaining to Arabic and other linguistically analogous languages. Our con- tributions can be summarized as follows:
Performance on Multilingual Tasks. In multi- lingual tasks, ChatGPT is found struggle with generating non-Latin scripts (Bang et al., 2023). but strategies such as prompting task descriptions in a high-resource language such as English can improve results (Lai et al., 2023). In line with this research, we find ChatGPT to work better with English prompts than Arabic prompts. Lai et al. (2023) evaluate ChatGPT in zero-shot setting on seven tasks covering 37 languages including Ara- bic. The authors show ChatGPT performs gener- ally well for high-resource languages such as En- glish, Russian, and German compared to medium- resource languages such as Arabic. Huang et al. (2023) introduce cross lingull thought and assess
1. We rigorously evaluate ChatGPT on a wide range of Arabic tasks and varieties, offering the first work benchmarking ChatGPT Ara- bic NLP at scale.
2. We contextualize ChatGPT by comparing it against BLOOMZ, another relatively large model (7.1B), and two finetuned dedicated Arabic models.
3. We perform systematic evaluation of ChatGPT and GPT-4 on dialectal Arabic (DA) as compared to MSA, revealing the weakness of both models on especially dialects.

---

the multilingual capability of ChatGPT on bench- marks. The authors specifically evaluate ChatGPT on Arabic for natural language inference task in few-shot settings. Compared to these concurrent efforts, our work conducts ChatGPT evaluation on Arabic at much larger scale with 44 diverse tasks, well as comprehensive analysis of ChatGPT and GPT-4's performance dialectal varieties. Wu al. (2023a) find that ChatGPT fails to surpass the passing score in Chinese medical licensing exam. Similarly, Kasai et al. (2023) show that ChatGPT fails to pass the Japanese medical licensing exam. our work, we conclusively show that ChatGPT is inferior to supervised models across a wide range of domains on several varieties of Arabic.
une as crensumtiny
are poruraate besanaer
zodkst lake
lonel
Soie ynd shae deg
loper eppur
She acsoourtoy, Your bul You
You prenaair leuoure andld Your
Yoa
de uib
Meticiie
Now podic Lod Sabmmng Aqua.
Figure Prompt templates for different tasks.
prompt can influence subsequent interactions with the model as well as its generated outputs by defin- ing set of specific rules. Therefore, in order to acquire desired outcome, it is important to clearly define the of rules and intents to the model. In our initial experiments, we experimented with a diverse set of prompt templates in both English and Arabic. Specifically, we evaluated on dialect iden- tification, machine-generatee text detection, and toxicity detection for NLU and machine transla- tion tasks for NLG with both English and Arabic prompts, observing that the English prompt outper- forms its Arabic counterpart. Hence, we design universal prompt template in English as follows: (1) We first set the role of the model (e.g., as an annotator for NLU tasks). (2) We provide name of the task that needs to be performed (e.g.. sen- timent classification). (3) We define what should be the expected outcome of the model (e.g., the label set for an NLU task). (4) If the task involves k-shot (k 0) learning, provide the model with examples. (5) Finally, we deliver test input the model produce the output. We present the templates of our designed prompts in Figure 2. We offer examples of k-shot NLU and NLG tasks in Appendix H.
3 Datasets
We evaluate ChatGPT on both Arabic NLU and NLG tasks. For NLU, we use a total of 48 datasets representing 20 different tasks from the ORCA benchmark (Elmadany et al., 2023). The diverse coverage of these datasets lends our evaluation broad representativeness and applicability to many real-world scenarios. For NLG, we compile 23 pub- licly available datasets from across 13 task clusters, We briefly introduce our evaluation datasets below and provide a detailed description of them in Ap- pendix B.
NLU Tasks, In NLU, we include 20 tasks span- ning across Arabic-NLI (1),² claim prediction (1), dialect identification (3), machine-generated text detection (1), paraphrase detection (1), sentiment analysis and emotion detection (2), social meaning detection (9), stance detection (1), and word sense disambiguation (1). Task-specific details with full citation of all datasets are in Appendix B.I.
NLG Tasks. We include 21 NLG tasks covering 13 task clusters: code-switched translation (2), di- acritization (1), dialect translation (6), grammat- ical eтror correction (1), MT (4), news title gen- eration (1), open-domain dialectal generation (3),
Experiments
We run experiments under 0-shot, 3-shot, 5-shot, and 10-shot settings. For the training data of the few-shot experiments, we randomly sample from each respective training dataset. For a ke-shot set- ting, we make sure that if a training sample is selected then it will also be selected for n-shot settings where n > k. For evaluation, we ran- domly sample set of 200 examples from the test set of each dataset, to keep the cost man-
paraphrasing (1), QA (1), question generation (1), text rewriting (1), transliteration (1), and summa- rization (1). Again, we provide task-specific details with appropriate references in Appendix B.2.
4 Prompt Design
A prompt is a set of instructions provided to an LLM that programs the model by enhancing its purpose and capabilities (White et al., 2023). A
'Comparison between the class distribution of the whole training set and the few-shot samples is in Appendix F.
Number of tasks in each cluster are in parentheses.

---

ageable. We evaluate ChatGPT (gpt-3.5-turbo), which is an optimized version of GPT-3.5 series, We set the temperature to 0.0 while generating re- sponses from ChatGPT. We compare this model with BLOOMZ (7.1B parameters), which is fine- tuned on a diverse set of tasks on 46 languages including Arabic (Muennighoff' et al., 2022b). We choose BLOOMZ since it is claimed to achieve im- pressive generalization
Emotion and Sentiment. ChatGPT achieves 17.33% F₃ on emotion detection with 5-shot Iearn- ing, while BLOOMZ achieves 17.13 with 10-shot. Both of these models, however, perform much lower than finetuned MARBERTy2 (F,=68.74) For sentiment, ChatGPT (64.96) outperforms BLOOMZ (50. 14) by margin of 14.52 F1 scores. Again, MARBERTvz (F1=68.92) outperforms both models.
unseen tasks. Fol- lowing Chung al. (2022a), we use free-form generation for both ChatGPT and BLOOMZ along with simple post-processing, c.g. removing lead- ing or trailing whitespace. Additionally, to fur- ther situate performance of ChatGPT, we fine- tune MARBERTvz (Abdul-Mageed et al., 2021a) and AraT5 model (Nagoudi et al., 2022b). Again, we choose these models they are reported to achieve SOTA on NLU and NLG in Abdul-Mageed al. (2021a) and Nagoudi et al. (2022b), respectively. For both MARBERTv2 and AraT5, we identify the best model on the respective development split (Dev). We train M2 for 25 epochs with a patience of 5 and AraT5 for 10 epochs. For both models, we set the learning rate to 5e-5 across all the tasks. We present our overall experimental setup in Figure 1. In addition reporting the per- formance of finetuned models on the same 200 set as ChatGPT and BLOOMZ, we report the 3-run aver- age of the models' performances on the respective full test sets (reported in gray). For NLU tasks, we use macro-F'1 scores and for NLG tasks we use the appropriate metric suited to each task (Table 2). We provide examples of NLG model responses in Appendix I.
Dialect. On binary-level dialect classification, ChatGPT achieves 77.00 F1 with 3-shot learning, outperforming BLOOMZ (49.66 6688 F1) by 27.32 points. Meanwhile, MARBERT (88.00 significantly out- performs both models. On region-level dialect, ChatGPT achieves 30.69 Fi, outperforming BLOOMZ (30.19). MARBERT outperforms both models (89.69). On country-level dialect, BLOOMZ exhibits lower performance (11.61) compared to ChatGPT (1) and yy (33.88). Similar to the two other dialect tasks, ChatGPT falls behind the much smaller model on country-level dialect ID.
Claim and -eneaae- Text. For claim verification, performance of ChatGPT in- creases with the increased number of shots (achiev- ing 61.82 F1). BLOOMZ scores less than half (30.87) compared to ChatGPT. ChatGPT falls be- hind MARBERTvz by margin of 3.71. On machine- generated text detection, ChatGPT and BLOOMZ achieve comparable performance (47.81 and 45.83, respectively). MARBERTv2 (76.23) outperforms both models by a large margin.
Toxic Text. On abusive language detection, the best score ChatGPT achieves is 42.03% (with 5- shot learning). This is close to BLOOMZ (42.26, with 5-shot). Again, MARBERTv2 is much better than both (82.96). On hate speech detection, ChatGPT achieves 50.99. It outperformed by BLOOMZ, which acquires 57.88 with 10-shot Iearn- ing. Both models are significantly outperformed by MARBERT (78.95). On offensive language, ChatGPT is at 73.76 (with 10-shots), and even its 0-shot outperforms all few-shots of BLOOMZ. Simi- lar to other tasks, MARBERT performs best (97.37). Given concerns about models like ChatGPT gener- ating toxic language, we also manually inspect its errors finding it to be sensitive to false toxicity (i.e.. tends to flag non-toxic text as toxic). Refer to Appendix D for our full toxicity analysis.
6 Evaluation on NLU Tasks
Overview. We present our evaluation on NLU tasks in Table I. We observe that although ChatGPT outperforms instruction-tuned multilin- gual models like BLOOMZ, the smaller finetuned model MARBERTv2 achieves significantly better re- sults than ChatGPT. We also find that model perfor- mance does not necessarily increase as we increase the number of shots. The common wisdom for this behavior is that improvement with few-shot learning is model- and task deeende,,,,,,,, and is often sensitive to the order of the shots Wei et al. (2021) )); Brown et al. (2020); Lu et al. (2022). We now discuss performance on each understanding task.
Irony and Sarcasm. On irony detection, 0-shot ChatGPT achieves 67.27. outperforming BLOOMZ
"Snapshot of gpt-3.5-turbo 3tturbo from March 1st 2023. 'https: / http//hgggggngggceeeee/ ca/bi wwwbbbicinneeeeeeeccceeeooeeeeee
'When we report a result without reference to the number of shots, it is typically the best result across all shots.

---

MARBERTyz (Test No.)
MARBERTv2
BLOOMZ (N-shot)
ChatGPT (N-shot)
Task
(Test No.)
3
10
10
200
Full
Dialect-Binary
44.47
49.68 09.68
66.98
.00
70.33
70.8
88. 00
86.91
Dialect-Region
2.31
20.28
30. 19
22.50
30.69
29.52
27.98
27. 17
89.69
06.32
Stalect-Country
0.81
12.34
12.84
14.61
13.86
18.60
19.74
19.43
33.88
36.06
Machine-Gen
2.64
29.08
45.83
45.83
26.
31.90
19.50
47.81
76.23
86.69
Abusive
13.21
23.24
42.26
42.10
36.90
34.98
42.03
36.60
82.96
78.03
Hate speech
9.01
52.61
51.06
57. 88
37 37.28
50.99
44.79
47.38
78.95
83.54
Offensive
12.78
60.01
47.48 48
37.94
60.32
71.61
69.34
73.76
97
92.23
Trony
13.44
54.81
52.51
52.51
59.51
57.37
83.09
Sarcasm
2.76
30.94
28, 28
29.41
42.58 58
41.71
43.99
43.09
80.10
76.19
Dungerous
45.07
17.40
22.48
18.97
24.35
27.98
26.97
37.
63.97
67.11
Adult
3.77
63.70
57.81
52.15
41.41
62.32
62.36
66. 12
93.95
90.97
Gender
7.14
40.48
35.52
36.32
1.02
8.18
27.36
65.04
64.97
67.64
Age
1,05
27.27
29.64
21.80
10.05
35.58
42.21
44.11
49.41
46.24
Claim
25.93
25.72
25.86
30.87
24.09
54.24
56.02
61.82
65.53
67.83
Emotion
11.40
9.66
17.13
14.54
17.33
16.58
70.82
Sentiment
43.72
45.53
50.00
50.44
58.00
62.37
64.96
63.18
68.92
80.83
Paruphrase
34.64
62.09
63.66
48.93
42.16
$2.45
82.97
85.80
65.98
63.47
Stance
14.53
23.85
24.49
21.66
61.51
62.22
66.60
63.70
88.23
80.57
XNLI
17.05
17.28
17.48
26.32
56.43
50.87
19.40
55.66
62.89
62.22
WSD
43.75
41.36
46.43
47.11
42.50
53.49
43.96
43.12
36.31
33.28
ORCA Score
15.64
35.27
37.06
36.21
37.98
46.29
46.87
51.28
68.85
71.00
Table 1: NLU Results. We report the Macro-F, score for every task. We evaluate BLOOMZ and ChatGPT in O-shot and in-context n-shot (where = 10) settings. MARBERTyz is our fully supervised model. We report the results of MARBERTyz on the same test set (200 samples) BLOOMZ and ChatGPT for fair comparison. The best scores are in bold. additionally report the performance MARBERT,z on full test set from Elmadany et al. (2022).
but again lags behind MV (85.83) by mar- gin of 18.56 points. On sarcasm, ChatGPT (43.99) outperforms BLOOMZ (30.94) but is almost half of M2RBERTv2 performance (80.10).
in BERT models, which we further discuss in Ap- pendix C.
Text-Pair Tasks. On paraphrase identification, ChatGPT with 10-shots (85.80) outperforms both BLOOMZ (63.66 with 5-shot) and MARBERTvz (65.98) by a large margin. Strikingly, ChatGPT with only 3- or 5-shots outperforms the fully-finetuned MARBERTv2 model. This shows the remarkable abil- ity of ChatGPT on semantic tasks such as para- phrase detection. On stance detection, all the few-shot setups of ChatGPT outperform BLOOMZ by significant margin. However, MARBERTyz (F1=88.23) outperforms ChatGPT. On natural lan- guage inference (XNLI), ChatGPT achieves F, of 56. 43 with 0-shot learning, while the best score BLOOMZ is at 26.32 (for 10-shots). Both models are outperformed by MARBERT (62.89).
Adult and Dangerous Content. ChatGPT achieves 66.12 Fi with 10-shot learning adult content de- tection, while BLOOMZ achieves at best 63.70 (with 3-shots). Aligning with the general trend thus far, MARBERTy2 (93.95) outperforms both models by significant margin. On the dangerous con- tent dataset, ChatGPT only achieves 37.11 Fj. In- terestingly, ChatGPT is outperformed by BLOOMZ (45.07) in 0-shot learning. However, ChatGPT dom- inates BLOOMZ in all other few-shot setups. Again, MARBERTy2 outperforms both models (63.97).
Demographic Text Classification. On age predic- tion, ChatGPT achieves 14.11 F, (with 10-shots), whereas BLOOMZ is at 29.64 (with 5-shots). Here, MARBERTy2 (49.41) outperforms ChatGPT by 5.30. On the gender task, ChatGPT (65.04) outperforms BLOOMZ (40.48). Also, ChatGPT performs better than MARBERTvz (64.97) by a slight margin.
7 Evaluation on NLG Tasks
Overview. We present the results of our evalua- tion on generation tasks in Table For NLG, we notice that ChatGPT performs better than BLOOMZ on the majority of tasks. However, following a similar trend to NLU, finetumed AraT5 consistently outperforms ChatGPT. We now present our results for each group of NLG tasks.
Word Sense Disambiguation. ChatGPT achieves the best score of 53.49 with 3-shots. The other few-shot settings of ChatGPT outperformed by the corresponding few-shot settings of BLOOMZ. Sur- prisingly, the finetuned model MARBERT vz is outper- formed by both ChatGPT and BLOOMZ by signif- icant margin. We suspect this is due to the issue of anisotropy (Ethayarajh, 2019; Li et al., 2020)
Text Rewriting and Paraphrase. For text rewrit- ing, BLOOMZ achieves 76.67 BLEU scores (with O-shots) which is better than ChatGPT results. Sur- prisingly, BLOOMZ'S performance deteriorates as we

---

Task
Metric
BLOOMZ (N-shor)
ChatGPT (N-sbot)
AraT5 (Test No.)
(Test No.)
AraT5
3
3
5
Full
Text Rewriting
BLEU
76.67
23. 73.96
13.97
12.73
11.39
58. 75
53.31
62.62
99.64
91.19
Puriphrase
BLEU
12:98
9.37
10.27
10.55
7.89
8.92
9.19
9.60
14,40
18.69
Question-Gen
BLEU
28.76
15.70
18.53
18. 69
14.48
19.86
20.08
18.15
35.17
33
QA
SQUAD
76.04
65.45
62.08
60. 49
32.96
51.73
54.
53.67
$1.45
83.34
Summarization
ROUGE-L
13.56
10.71
88
20.43
35,31
26
0.99
0.79
3.24
4.72
4.62
Diacrilizatinn 1
CER
0.51
1,33
L62
1.42
0.11
0.06
0.05
0.01
Transliteration.
CER
0.59
0.45
0.42
0.42
0.27
0.24
0.24
0.23
0.18
0.18
MI (en-+a57
BLEU
5.33
12.54
12.35
10.07
20.32
23.38
23.31
23.71
27.12
28 12
MT tes-aa))
BLEU
6.94
9.20
9.31
7.33
16.47
18.11
17.45
19:32
21.16
21.74
MI (i)+aa))
BLEU
6.88
5.51
5.76
4.97
15.12
15.44
15.37
16:26
18.48
20.31
MT
BLEU
2.42
1.95
15.83
17.35
(J-en-ten)
10.91
37.38
CST (Dx-fe-ffr)
BLEU
28.41
28.27
26.75
28.61
34.61
35
36.45
37.95
17.49
16.16
GBC
M Scoorr Fuz
2.40
1.12
2.
18. 72
72
46.56
67.54
70.51
Table 2: NLG Results. Higher is better unless otherwise specified by 4. We evaluate BLOOMZ and ChatGPT in O-shot and in-context n-shot (where = 5, 10) settings. AraT5 our fully supervised model. The best scores in bold. QA Question Answering. MT Machine Translation, CST Code Switched Translation. We report the results of AraT5 on the same test set (200 samples) as BLOOMZ and ChatGPT for a fair comparison.
increase the number of training examples whereas ChatGPT shows best performance (62.62) with 10- shots. However. both models are significantly dominated by AraT5. For paraphrase generation, BLOOMZ outperforms ChatGPT in all k-shot setups. Noticeably, the performance of ChatGPT monoton- ically improves with the increased number of shots. AraT5 outperforms both ChatGPT and BLOOMZ with a BLEU of 14.40.
1, which means it mistakenly inserts additional characters during prediction. Although ChatGPT (CER=0.05) exhibits impressive performance on this task, AraT5 (CER=0.03) still outperforms it. For transliteration, ChatGPT (0.23) again outper- forms BLOOMZ (0.42). However, AraT5 (0.18) achieves even lower error rates than both.
Machine Translation. ChatGPT outperforms BLOOMZ on all the X (English, Spanish, French, Russian) -> Arabic MT tasks. As expected, both models perform better when English is used as the source language. Although the performance gap is smaller in MT as compared to the general trend in other tasks, AraT5 is still better than ChatGPT.
Question Generation and Question Answering. For question generation, ChatGPT is outperformed by 0-shot performance of BLOOMZ. Nevertheless, unlike ChatGPT, BLOOMZ performance does con- sistently improve as we increase the number of training shots. Compared to AraT5, both ChatGPT and BLOOMZ exhibit significantly lower scores. For QA, BLOOMZ significantly outperforms ChatGPT in all the few-shot settings. Specifically, BLOOMZ achieves the best score of 76.04 with 0-shot learn- ing, whereas ChatGPT achieves 54.14 at best with 5-shot learning. However, both models are out- performed by AraT5 (81.45). We suspect BLOOMZ performs well on QA since it has been explicitly finetuned on this task using Arabic data.
Code-Switched Translation. For Jordanian Ara- bic (Jo) mixed with English - English and Al- gerian Arabic (Dz) mixed with French -> French code-switched translation tasks, ChatGPT sig- nificantly outperforms BLOOMZ by 10-30 points. ChatGPT performs slightly better for mixed En- glish + English, while BLOOMZ performs better in mixed French - French than mixed English >> English translation. Interestingly, both ChatGPT and BL BLOOMZ show highly superior performance than the finetuned AraT5. We suspect that both models have been pretrained with a lot of data from high- resouree languages like English and French. This pretraining step helps them to perform better than the Arabic-dedicated smaller model.
Summarization and News Title Generation. For summarization, ChatGPT achieves 20.43 ROUGE, outperforming BLOOMZ by a margin of 6.87 points. Both ChatGPT and BLOOMZ are also outperformed by AraT5. For news title generation, ChatGPT is at 4.72 BLEU points whereas BLOOMZ struggles (1.0
Grammatical Error Correction. We present Scorer Fo.s score (Dahlmeier and Ng. 2012) for GEC in Table Due the longer sequence length (> 4, 096), we exclude 10-shot evaluation for this task. We find ChatGPT to significantly outper- form BLOOMZ (48.72 vs. 2.40 Fus score). However,
BLEU). Again, AraT5 better than both (7.72). Diacritization and Transliteration. ChatGPT dominates BLOOMZ with significantly lower error rates for diacritization. BLOOMZ even struggles to keep the character error rate (CER) lower than

---

AraT5 (67.54) is much better than both.
MSA data than DA at some stage of their develop- ment and (ii) GPT-4 performs generally superior than ChatGPT in both MSA and DA samples.
8 Performance on Dialectal Arabic
DA has traditionally been only spoken. It only became easier to collect dialectal data with the proliferation of social media where these varieties started to be used. Regardless, most Arabic dialects remain mnder-investigated due to rarity of resources. Motivated by this, we dedicate the current section to investigate performance of ChatGPT on DA as compared to MSA. In the context of this investiga- tion, we also compare ChatGPT and GPT-4 on these two broad categories of Arabic.
Dialectal NLG. To evaluate ChatGPT on dialectal generation, we run it on MSA and five Arabic di- alects + English MT tasks from the Multi-dialectal Parallel Corpus (MDPC) proposed by Bouamor et al. (2014). Namely, we use data involving Egyp- tian, Jordanian, Palestinian, Syrian, and Tunisian.
For the current analysis, we first select tasks labeled involving more DA in ORCA (i.e., all 11 tasks in Figure 3). We then run a strong in-house MSA vs DA classifier (~~ 88% F1) to separate MSA and DA samples, keeping only samples predicted with 80% confidence. This gives us an average of 119 DA and 78.82 MSA samples across all 11 tasks. We then evaluate the macro-F; performance of ChatGPT and GPT-4 on these selected samples. Dialectal NLU. To identify the extent which GPT-4 can detect country-level dialect, we run it on our test data under O-shot. We find GPT-4 to achieve 26.92 Fi, which is 13.06 improvement over ChatGPT performance reported in Table 1 (i.e, 13.86). As shown in Figure 3, both ChatGPT and
(a) K-shot BLEU scores of ChatGPT on MSA and 5 di- alects - English MT.
(b) Zero-shot BLEU sores of ChatGPT and GPT-4 on MSA and 5 dialects + English MT.
Figure 4: ChatGPT and GPT-4 on dialectal MT.
As Figure 4a shows, ChatGPT MSA performance is better than its dialectal performance on all our k-shot settings. This further substantiates what our NLU results above suggest as to ChatGPT's MSA and DA abilities. Comparing results across dialects, ChatGPT performs best on Egyptian, which may be directly due availability of Egyptian dialectal data online as compared to rarity of data from other dialects (Nagoudi et al., 2022a).
100
ChaiGrt UEAA
GaTe (M52))
Chuiooi Ipwi
ChatGPT compared to GPT-4. We carry out an ad- ditional set of experiments to compare ChatGPT and GPT-4 on the same five dialects and MSA test sets from Bouamor et al. (2014) listed above, but subsampling only 50 data points from each and reporting both models in zero-shot over the sub- sample. As Figure 4b shows, for MSA and all dialects except Jordanian, GPT-4 still outperforms ChatGPT. We also notice that GPT-4 wins with a large margin on dialects such as Palestinian, Syrian, and Tunisian, all of which are on the low-resource side compared to dialects such Egyptian. This finding suggests that, compared ChatGPT, GPT-4 may have seen more Arabic varieties, and perhaps more data from some of the varieties.
aline (Da)
60
10
sprsinn speetc
Halo
oftecion
bungertus
Adut
Gender
Inotzon
Strtenent
bottace
Figure 3: Comparison between ChatGPT and GPT-4 on MSA vs DA in macro-F, for 11 ORCA tasks.
GPT-4 perform better on MSA compared to DA (on 8 and 9 out of the 11 tasks, respectively). It is also clear that GPT-4 outperforms ChatGPT on both MSA and DA for the majority of the tasks (9 tasks for MSA and tasks for DA). On average, GPT-4 improves over ChatGPT by 18. 18.62% for MSA and 10.40% for DA tasks. This finding suggests that (i) both ChatGPT and GPT-4 -4 have likely seen more
Human Evaluation
We also perform a set of human evaluations, moti- vated by the potential of human evaluation to cap- ture subtlcties of language that automated metrics may overlook. We carry out our evaluation on eight

---

Human
GPT-4
Taplee
cerote
DC
Do
BE.0MZ
CheeCiPT
GPT4
BI BLOOMZZ
CherGiT
CPPT
Figure 5: Evaluation of the models' responses by human and GPT-4. 4. A is the best, and D the worst rating. MT Machine Translation, CST Code Switched Translation, DG Dialogue Generation.
NLG tasks: code-switched translation (2), dialogue generation (3), machine translation (I), paraphrase (I), and summarization (1). We particularly pick these tasks as they represent diverse generation task categories and ChatGPT performs poorly some of them in our automatic evaluation setting (Table 2).
ity using diagnostic test cases. Specifically, we manually evaluate ChatGPT 's capability of En- glish mixed MSA, Egyptian, and Moroccan code- switched translation generation from plain English text. We ask annotators who are fluent the respective varieties, as well English, to evaluate diagnostic dataset based on fluency (defined as how fluent the translated text is), faithfulness (de- fined as how semantically close the translated text is to the source text), and code-switching ability (defined as how accurately the translated text in- cludes code-switching). We present our result in the Table 3.
Evaluation Setup. We build our human evaluation framework on top of Wang et al. (2022b), Wu et al. (2023b) and implement a four-level (A, B, C, D) rating system that we adapt to language generation. (Refer to Appendix G.1 for details). We prepare the framework with examples and ask three pairs of native Arabic speakers (total=6) to rate 50 samples each from the output of each task, based on how effectively a sample has fulfilled the task described in the input prompt.
We observe that ChatGPT produces fluent trans- lations that are also semantically close (faithful) to the source text. However, ChatGPT struggles to produce code-switched text (i.e., it generates mostly in Arabic script). Interestingly, this issue is more prevalent for MSA than Egyptian and Mo- roccan. We hypothesize that this discrepancy cuts across several linguistic categories and involves top- ics such as translation of endearment expressions; mulli expressions, idioms, and proverbs; negation; sub-token-level code-switching: and di- alectal variants of MSA lexica We further suspect that the tokens in the source English text are very common in MSA. As result, the model does not seem able to code-switch the words into English.
Results. We aggregate scores from each pair of an- notators and use Cohen's kappa metric to measure the inter-annotator reliability. We find the inter- annotator agreement more than 0.6 for the majority of the tasks. We report the result in Figure (Refer to Appendix L for more detailed results). From human evaluation, we find that outputs produced by GPT-4 and ChatGPT, for all tasks, are rated as mostly of fine quality. We also find that ChatGPT is rated higher than BLOOMZ for summarization in the human evaluation than it is in the automated setting, perhaps reflecting that, although useful, ROUGE is not ideal metric for summarization as it captures only token-level overlap rather than deep semantics. addition, ChatGPT is rated higher than BLOOMZ in human evaluation than in automated evaluation for paraphrasing. This, again, reflects BLEU not being ideal for paraphrase detection for the same reasons as in the case of summarization.
10 Using GPT-4 to Evaluate Responses
Similar to Chiang et al. (2023) and Zhou et al. (2023), we assess the quality of the generated re- sponses from ChatGPT, BLOOMZ, and GPT-4 using the GPT-4 model itself. For this purpose, we de- sign prompt (Figure 8 in Appendix E) that takes the input and the corresponding responses from the models and asks GPT-4 to rate between 'A' and 'D' (where 'A' is the highest) for each response. (Refer to Appendix J and Appendix K for illustrative sam- ples). We do this analysis using a random sample of
Human Analysis on CST. As evident from Ta- ble 2, ChatGPT outperforms the finetuned model on the code-switched translation task. This prompted us to further probe ChatGPT code-switchin abil-

---

fluency
faithfulness
code-switching
Annotator
CST
(A/B/C/D)
(A/B/C/D)
(A/B/C/D)
Annotator
En->MSA-En
80/10/10/0
80/20/0/0
0/40/0/60
En->Egy-Enn
70/10/20/0
100/0/0/0
20/30/0/50
En->Mor-En
10/40/50/0
90/0/10/0
20/30/0/50
Arg
53.3/20/26.700
90/6.7/3 7/33/0
13.3/33.3/0/533
Annotator
En->MSA-En
30/70/0/0
90/0/10/0
0/30/0/70
En->Egy-En
50/40/10/0
90/10/0/0
0/40/0/60
En->Mor-En
20/40/40/0
100/0/0/0
20/30/0/50
Avg
33.3/50/16.7
93.4/3.3/3.3/0
6.7/33.3/066
Average
43.3/35/21.7/0
91.7/5/3.3/0
10/33.3/0/56.7
Table 3: Human evaluation of ChatGPT's responses on the three code-switched translation tasks.
25 data points from each of the datasets evaluated by human annotators cited in Section 9 above." & As Figure 5 shows, GPT-4 provides more accurate and satisfying responses (compared to ChatGPT and BLOOMZ), followed by ChatGPT, almost all cases. BLOOMZ is rated 'D' on all of the tasks. Upon in- specting the explanations from GPT-4, we find that this poor rating of BLOOMZ is mostly due to it just copying the input samples rather than following the input instruction properly.
Figure Human and GPT-4 agreement 8 NLG tasks. On average, at least one human evaluator agrees with GPT-4 evaluation 71.5% of the time.
Does GPT-4 eval correlate with human eval? Chiang et al. (2023) and Zhou al. (2023) show a positive correlation between human and GPT-4 evaluation, which we also for Arabic NLP in this work. Hence, we compute the percentage of evaluation agreement between humans and GPT-4 on the models' responses, More specifically, for each task, we calculate at how many occurrences human evaluators agree with GPT-4 evaluation. As Figure 6 shows, at least one human evaluator agrees with GPT-4 evaluation 71.5% of the time aver- age. This suggests that may be possible use GPT-4 automate the time-consuming and costly process of using humans for assessing model per- formance. This is at least true for the tasks we consider.
uation covers both NLU and NLG under k-shot set- tings. Comparing results from ChatGPT to BLOOMZ and finetuned models, we show that ChatGPT, in spite of its large size, can struggle on multiple Ara- bic tasks compared to much smaller finetuned mod- cls. We conduct extensive quality assessment using both human and GPT-4 evaluation of model responses and find positive alignment between human and JPT-4judgment We further investigate the performance of ChatGPT and GPT-4 on MSA vs DA tasks, revealing inherent limitations of both models, particularly when applied to dialects. Our findings accentuate the inherent multilingual limita- tions observed in both ChatGPT and GPT-4, thereby indicating substantial scope for improving these models. We intend expand evaluations encompass additional low-resource languages.
11 Conclusion
We presented a comprehensive evaluation of ChatGPT on 44 diverse Arabic NLP tasks. Our eval-
12 Limitations
"This gives us 25 samples x 8 datasets = 200 decisions, for which we use the OpenAl playground. At the time of writing this paper, OpenAl allowed only 25 examples per hours on GPT-4 playground.
Experimental Setup. this work, we evaluate with a randomly selected subset of test samples

---

keep the cost manageable. Although the ran- dom selection should idcally represent the whole distribution, the performance may vary slightly when comparing to the whole test set. Addition- ally, ChatGPT's version can be updated on a regular interval. Hence, the results and the analyses re- ported here should be treated accordingly, since the model's responses can change over time (Chen et al., 2023a). To save time and cost, we perform 4PT-4 generation and human, GPT-4 evaluation on a diverse but selective tasks, which we wish to ex- tend to the other tasks in the future.
we collect from 23 publicly available datasets. To ensure proper credit assignment, we refer users the original publications (Appendix B).
Intended Use. We believe our work will spur fur- ther research on studying LLMss on Arabic NLP benchmarks. As our findings show, the existing multilingual LLMs still lag behind compared to the smaller finetuned models the majority of Ara- bic NLP tasks. Therefore, our work can arise the interest among the researchers to develop Arabic language dedicated L.MSs that can match or out- perform the SOTA finetuned models.
Model Variation. We perform evaluation on three multilingual LLMs, namely, ChatGPT, GPT-4, and BLOOMZ. Additionally, we finetune two Arabic lan- guage dedicated SOTA models i.e., yy and AraT5. To keep our work resource-cfficient, we do not include other massive LLMs such as PaLM 5408 (Chowdhery et al., 2022). Also, since we are already comparing against the finetuned SOTA models (MARBERTyz and AraT5), we exclude other multilingual models like mT5 (Xue et al., 2021). However, acknowledge that incorporating large number of models can facilitate the compara- tive analysis.
Potential Misuse and Bias. Since there exists lit- te to no clarity on the training data of ChatGPT and GPT-4, these LLMs can produce potentially harmful and biased contents (Laskar et al., 2023). Therefore, we recommend that these models not be used in applications without careful prior consider- ation of potential misuse and bias.
A Acknowledgments
We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineer- ing Research Council of Canada (NSERC; RGPIN- 2018-00 88-04667), the Social Sciences and Humanitics Research Council of Canada (SSHRC; 435-2018- 0576; 255-2000-10044 Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada, and UBC Advanced Research Sonputing-Sockeyer 10
Model Evaluation. On open-domain dialectal di- alogue generation tasks, we empirically find that all the models perform poorly (close to 0.0 BLEU) with the automatic metric. Since these tasks re- quire the responses be fluent and coherent and not particularly follow any closed form (c.g., MT tasks), would not be fair evaluate the models using such automated metrics. Therefore, we ex- clude the dialectal dialogue generation tasks from the automatic evaluation (Section 7) and conduct both human (Section 9) and GPT-4 evaluation (Sec- tion 10) on them.
Canadian
References
Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic Dialect Identification the Wild. Proceedings of the Sixth Arabic Natural Language Processing Work- shop.
Findings. Some of our findings suggest that GPT-4 can be employed to automate the process of eval- uating model responses, thus replacing humans. Although this can accelerate development of AT systems and deploying them for decision making, it might also have negative implications on work- force. In particular, we recommend that our find- ings should be couched with caution and not be considered as a reference replace workforee with AI.
Muhammad Abdul Abdul-Mageed, AbdelRahim Elmadany. and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Ara- bic. In Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 7088-77 1088-7105, Online. Association for Computational Linguistics.
Muhammad -bgul-M AbdelRahim Elmadany. and El Moatez Billah Nagoudi. 2021b. ARBERT & MARBERT: Deep bidirectional transformers for Ara- bic. In Proceedings the 59th Annual Meeting of the
13 Ethics Statement
Data Collection and Release. We collect the NLU evaluation datasets from ORCA. For NLG tasks,
"htps://alliaacccaa.cc https::/aarc.uubb wwwpaa/wwwauuuarrcoocceeeee

---

Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 7088-7105, Online. Association for Computational Linguistics.
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 21-27.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji. Tiezheng Yu, Willy Chung. Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
Muhammad Abdul-Maaeed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natu- ral Language Processing Workshop, pages 97-110, Barcelona, Spain (Online). Association for Computa- tional Linguistics.
Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of arabic. In LREC, pages 1240-1245.
Muhammad Abdul-Mageed, Chiyu Zhang. Azadeh Hashemi, and EI Moatez Billah Nagoudi. 2020b. AraNet: A deep learning toolkit for Arabic social media. In Proceedings of the 4th Workshop on Open- Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 16-23, Marseille, France. European Language Re- source Association.
Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic ine-grained di- alect identification. In Proceedings of the Fourth Ara- bic Natural Language Processing Workshop, pages 199-207.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020, Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Ibrahim Abu Farha and Walid Magdy. 2021. Bench- marking cansformer-basee language models for Ara- bic sentiment and sareasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. pages 21-31, Kyiv, Ukraine (Virtual). As- sociation for Computational Linguistics.
Ali Salch Alammary. 2022. Bert models for arabic text classification: A systematic review. Applied Sciences, 12(11).
Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. User-centric gender rewriting. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 618-631, Seattle, United States. Association for Computational Linguistics.
Lingjaao Chen, Matci Zaharia, and James Zou, 2023a. How is chatgpt's & behavior changing over time? arXiv preprint arXiv: 307009009....
Xuanting Chen, Junjie Ye, Can Zu. Nuo Xu, Rui Zheng. Minlong Peng. Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023b. How robust is gpt-3,5 to predecessors? a comprehensive study on language understanding tasks.
Ali Alshehri, EI Moatez Billah Nagoudi, and Muham- mad Abdul-Mageed. 2020. Understanding and de- tecting dangerous speech in social media. In Pro- ceedings of the 4th Workshop on Open-Source Ara- bic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 40-47, Marseille, France. European Language Resource As- sociation.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E, Gonzalez, lon Stoica, and Eric P. Xing, 2023. Vicuna: An open- source chatbot impressing 4 with 90%* chatgptt quality.
Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2019. Anetac: Arabic named entity transliteration and classification dataset, arXiv preprint 1:XXi::900.0311 10.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint S.Ki:2204.023114
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross- lingual Transferability of Mono- lingual Representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4637.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha
Ramy Baly, Mitra Mohtarami, James Glass, Lluís Marquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact check- ing in a unified corpus. In Proceedings of the 2018

---

Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022a. Seal- ing instruction-iiettneed language models. ArXiv, a0s/2210. 1416.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, EL,M,, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNL.P-IJCNIPP) pages 55-65, Hong Kong. China. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022b. Scaling instruction instruction-finetuned language models.
Ali Fadel, Ibraheem Tuffaha, Bara' Al-Jawarnch, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritiza- tion using deep neural networks.
Ibrahim Abu Farha and Walid Magdy. 2020. From Ara- bic Sentiment Analysis Sarcasm Detection: The ArSareasm Dataset. In Proceedings of the 4th Work- shop on Spprce Arabic Corpora and Process- ing Tools, with a Shared Task on Offensivve Language Detection, pages 32-39.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ima Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguistics.
Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: An empirical study.
Bilal Ghanem, Jihen Karoui, Farah Benamara. Véronique Moriceau, and Paolo Rosso. 2019.
IDAT @ FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets. In Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR- WS.org, Kolkata, India, December 12-15.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572. Montréal, Canada. Association for Compu- tational Linguistics.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171-4186.
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. So- hel Rahman, and Rifat Shahriyar. 2021. XI-sum: Large-scale multilingual abstractive summarization for 44 languages.
Amr Hendy, Mohamed Abdelrehim. Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? a comprehensive evaluation.
Mahmoud El-Haj, 2020. Habibi-a multi dialect multi national arabic song lyrics corpus. In Proceedings of the 12th Language Resources and Evaluation Con- ference, pages 1318-1326.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages created equal in Ilms: Improving multilingual capability by cross-lingual- thought prompting.
Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, I 166):2567.
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Mageed.. 2022. Orea: A challeng- ing henchmark for arabic language understanding. ArXiv, abs/22 12.
Wenxiang Jiao, Wenxuan Wang. Jen tse Huang, Xing Wang, and Zhaopeng Tu, 2023. Is chatgpt a good translator? yes with gpt-4 as the engine.
Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluating gpt-4 and chatgpt on japanese medical licensing ex- aminations.
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Mageed. 2023. Orea: A challeng- ing benchmark for arabic language understanding.

---

Jude Khouja. 2020. Stance prediction and claim veri- fication: An Arabic perspective. In Proceedings of the Third Workshop on Fact Extraction and VERifica- tion (FEVER), pages 8-17, Online. Association for Computational Linguistics.
Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wa- jdi Zaghouani. and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39-47, Doha, Qatar. Association for Computational Linguistics.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large lan- guage models in multilingual learning.
Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of OSACT4 Arabic offensive language de- tection shared task. In Proceedings of the 4th Work- shop on Open-Source Arabic Corpora and Process- ing Tools, with a Shared Task on Offensive Language Detection, pages 48-52, Marseille, France. European Language Resource Association.
Md Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan. Shafiq R. Joty, and J. Huang. 2023. A systematic study and compre- hensive evaluation of chatgpt on benchmark datasets. ArXiv, abs/2305. 18466.
Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2021. Adult content detection Arabic Twitter: Analysis and experiments. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 136- 144, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Bohan Li. Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 91 19-9130, Online. Association for Computa- tional Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika,
Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hai- ley Schoelkopf. Xiangru Tang. Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Alhert Webson, Edward Raff, and Colin Raffel. 2022a. Crosslingual generalization through multitask finetuning.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-tumn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Lnguage Processing (Volume I: Long Papers), pages 986-995.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts. Stella Rose Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir R. Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022b. Crosslingual gen- eralization through multitask finetuning. ArXiv, abs/2211.01788
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual language models.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint 1rXiv: 1192.
El Moatez Billah Nagoudi, Muhammad MageldM AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2022a. Jasmine: Ara- bic gpt models for few-shot learning. arXiv preprint arXiv:2212. 10755.
Yao Lu, Max Bartolo, Alastair Moore, Schastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers). pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed, 2022b. AraT5: Text- to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628-647, Dublin, Ireland. Association for Computational Linguisties.
El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. Machine generation and detection of Arabic manipu- lated and fake news. In Proceedings of the Fifth Ara- bic Natural Language Processing Workshop, pages 69-84, Barcelona, Spain (Online). Association for Computational Linguistics.
Saif Mohammad, Felipe Bravo-Marquezz Mohammad Salameh, and Svetlana Kiritchenko. 2018 SemEval- 2018 task I: Affect in tweets. In Proceedings of The 12th International Workshop Semantic Evaluation, pages 1-17, New Orleans, Louisiana. Association for Computational Linguistics.

---

Tarek Naous, Zahraa Bassyoumi. Bassel Mousi, Hazem Hajj. Wassim El Hajj. and Khaled Shaban. 2023. Open-domain response generation in low-resource settings using self-supervised pre-training of warm- started transformers. ACM Transactions on Asian and Low-Resoure Language Information Process- ing, 22
Yiming Tan, Dehai Min. Yu Li, Wenbo Li, Nan Hu. Yongrui Chen, and Guilin Qi. 2023. Evaluation of chaigpt a question answering system for answering complex questions.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elhayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez. Prangthip Hansanti, John Hoffman, Semarley Jarrett. Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Saftyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing.
Reham Omar, Omij Mangukiya, Panos Kalnis, and Es- sam Mansour. 2023, Chatgpt versus traditional ques- tion answering for knowledge graphs: Current status and future directions towards knowledge graph chat- bots.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding.
Boxin Wang, Chejian Xu, Shuohang Wang. Zhe Gan, Yu Cheng. Jianfeng Gao. Ahmed Hassan Awadal- lah, and Bo Li. 2022a. Adversarial glue: multi- task benchmark for robustness evaluation of language models.
Keqin Peng. Liang Ding, Qihuang Zhong, Li Shen, Xueho Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang. Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie, 2023a. On the robustness of chatgpt: An adversarial and out-o--distribution perspective.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023a. Is chatgpt a general-purpose natural language process- ing task solver?
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023b. Is chatgpt general-purpose natural language process- ing task solver? CoRR, abs/2302.06476.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022b. Self-instruet: Aligning lan- guage model with self generated instructions. ArXiv, abs/2212.
Michael V. Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: cautionary remark.
Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng. and Rui Xia. 2023b. Is chatgpt a good sentiment analyzer? a preliminary study.
Yves Scherrer. 2020. TaPaCo: A corpus sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 6868-6873, Marseille, France. European Language Resources Association.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Haitham Seelawi, Ahmad Mustafa, Hesham Al- Bataineh, Wael Farhan, and Hussein T Al-Natsheh. 2019. Nsurl-2019 task 8: Semantic question simi- larity in arabic. In Proceedings of The First Inter- national Workshop on NLP Solutions for Under Re- sourced Languages (NSURL 2019) co-located with ICNLSP 2019-Short Pupers, pages 1-8.
Jules White. Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineer- ing with chatgpt. ArXiv, abs/2302. 13382.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback.
Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. In chatgpt we trust? measuring and characterizing the reliability of chatgpt.

---

Jiageng Wu, Xian Wu, Zhaopeng Qiu, Minghui Li. Yefeng Zheng, and Jie Yang. 2023a. Qualifying chi- nese medical licensing examination with knowledge enhanced generative pre-traiming model.
A Literature Review
A.1 Machine Translation
Language models trained large-scale multilin- gual data have proven effective for wide range of tasks spread across multiple languages. Jiao et al. (2023) evaluate ChatGPT for machine translation (MT) tasks, reporting that ChatGPT's performance on MT is on par with commercial translation tools such as google-translate. However, they also find that when translating into a distant language pivor prompting is very effective. In pivot prompting. instead of directly translating the source into the target, first translating the source into high source similar to the target language and then trans- lating into the target followed. Further, Jiao et al. (2023) notice that for domain-speeift translation (e.g., in the biomedical filed), ChatGPT's perfor- mance degrades considerably. Peng et al. (2023) find that ChatGPT performs reasonably well in high- resource and domain-spectin settings but providing additional information can be vital. Gao et al. (2023) corroborate observations of Peng al. (2023) by designing prompts that include infor- mation such as domain, finding it to improve the MT performance of ChatGPT. Extensive evalua- tion by Hendy et al. (2023) shows that ChatGPT is quite good for translating into high-resource tar- get languages but its performance degrades for low-resource languages. Zhu al. (2023) evalu- ChatGPT and other LLMs such XGLM (Lin et al., 2022), OPT (Zhang et al., 2022), and BLOOMZ (Muennighoff et al., 2022a) showing that even though ChatGPT the best zero-shot and in-context few-shot model considered, it lags be- hind full supervised no language left behind models (NLLB) (Team et al., 2022).
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muham- mad Abdul-Mageed and Alham Fikri Aji. 2023b. Lamini-Im; diverse herd of distilled models from large-scale instructions.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 443.499,00 On- line. Association for Computational Linguistics.
Omar F Zaidan and Chris Callison-Burch 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171-22)
Susan Zhang, Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.07063
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why does chatgpt fall short in answering ques- tions faithfully?
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023a. Can chatgpt understand too? comparative study chatgpt and fine-tuned bert. ArXiv, abs/2302.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023b. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.
Chunting Zhou, Pengfei Liu,, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis. Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less more for alignment.
A.2 QA
ChatGPT demonstrates impressive results on prestion-answering tasks including user queries where no context provided. Zheng et al. (2023) evaluate ChatGPT on complex open-domain QA tasks, studying the failures of ChatGPT and propos- ing methods to improve the faithfulness of its an- swers. Tan et al. (2023) showcase ChatGPT's 's lim- itations the require math and reasoning skills. They show that Chat tGTT is far behind the fully supervised state-of- art models on these reasoning tasks. Further evalu- ations of ChatGPT by Shen et al. (2023) on a wide range of QA tasks demonstrate ChatGPT's perfor-
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empiriaal results and analysis.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? CoRR, abs/2305.03514.
ienteedde-intnssiee tasks which
Michal Ziemski, Marcin Dowryss,Dwwuut,, and Bruno Pouliquen. 2016. The united nations parallel corpus vl. O. In Lrec.

---

mance across various domains. The authors show that ChatGPT underperforms on domain-specific QA and is also very susceptible to adversarial ex- amples and perturbation. Omar et al. (2023) eval- uate ChatGPT for QA on knowledge graphs. They notice that even though it falls behind the super- vised SOTA methods, it can be robust QA system for knowledge graphs.
tion labels (e.g.. anger in emotion detection). Fur- thermore, it is observed that ChatGPT exhibits a tendency to predict a neutral label that is easily recognized in the colloquial language (e.g., stereo- type in the hate speech detection task), instead of utilizing a more precise label derived from the pro- vided taxonomy (e.g., white grievance), Qin et al. (2023b) evaluate ChatGPT on 20 NLP datasets en- compassing seven task categories in zero-shot set- ting. Although ChatGPT performs less effectively than models fine-tuned specifically for each task, it demonstrates superior reasoning capabilities com- pared to other instructed finetuned models (e.g., FLAN) in natural language inference, arithmetic reasoning, and QA tasks.
A.3 Text Classification
Text classification one task where ChatGPT does exceptionally well in zero-shot and in-context few- shot settings. It is often even on par with full- supervised models. Zhong et al. (2023b) evalu- ate ChatGPT on GLUE NLU benchmark (Wang et al., 2019) and compare it against fully supervised BERT (Devlin al., 2019) and RoBERTa (Liu al., 2019) baselines. Authors conclude that ChatGPT outperforms BERT and RoBERTa on MNLI, SST2, and RTE while underperforming these models on other GLUE tasks. Further, ChatGPT's evaluation by Gilardi et al. (2023) shows that it can outperform well-trained human anno- tators and crowd-workers for text classification tasks such as relevance, stance, topics, and frame detection. Wang et al. (2023b) test ChatGPT a wide range of sentiment analysis datasets and find that zero-shot ChatGPT on par with the fully supervised BERT model but lags behind the SOTA. The non-determiiistic nature ChatGPT prompted Reiss (2023) to assess the reliability and consistency of ChatGPT for text classification tasks in the zero-shot setting. They argue that zero-shot outputs for text classification do not meet the sci- entific threshold of reliability. The authors also find that minor alterations in prompts can change ChatGPT output. They recommend pooling the out- puts from multiple repetitions to improve reliabil- ity.
Robustness and Generalization. ChatGPT does well on a wide range of downstream NLP tasks of- ten yielding SOTA results if prompted properly. However, several studies reveal that its perfor- mance downgrades on domain-specialize tasks and is very susceptible to adversarial examples. Wang et al. (2023a) attempt to test the robustness and generalization capability of ChatGPT. The au- thors evaluate ChatGPT on AdvGLUE Wang et al. (2022a) and ANLI Nic et al. (2020) for adversarial robustness. They perform an evaluation on Flip- kart Review and DDXPlus medical diagnosis for the out-of-distribution (OOD) generalization. The results show that ChatGPT outperforms all consid- ered models (including zero-shot and fully super- vised SOTA models) on every task in both settings, Chen et al. (2023b) observe on average 35.7 7.1% and 43.59% performance drop in NLI and senti- ment analysis tasks, respectively, which further highlights ChatGPT's vulnerability to challenging and complex scenarios.
A.4 Multilinguality
ChatGPT is adept at generating high-quality re- sponses user queries. also demonstrates im- pressive capabilities in multiple languages. Lai et al. (2023) evaluate ChatGPT on seven different tasks and 37 different languages belonging to low, medium, and high resource families. The results show that ChatGPT is either at par or even better in some tasks compared to fully supervised SOTA baselines, particularly for the high-resource lan- guages. They also observe that for the low-resource language, providing task descriptions in a high- resource language can be helpful. The work on the multilingual evaluation of ChatGPT by Bang et al.
Ziems al. (2023) investigate the zero-shot abil- ity of ChatGPT across a broad spectrum of com- putational social science benchmarks encompass- ing various subject areas including sociology, psy- chology, literature, history, linguistics, and politi- cal science. They find that ChatGPT demonstrates poor performance tasks characterized by struc- tural complexity (e.g., event arguments) or those entailing subjective expert taxonomies (e.g., hate, empathy). However, ChatGPT attains high perfor- mance on tasks that involve either objective ground truth (like fact-checking tasks) or explicit defini-

---

(2023) establishes that it is better understand- ing non-Latin scripts but poor at generating them. Huang et al. (2023) propose cross-lingual thought prompting to improve the multilingual capabilities of ChatGPT and other LLMs. The experiments by Wu et al. (2023a) show that through careful ex- ploitation of domain knowledge, ChatGPT can out- perform the average human score on the China Na- tional Medical Licensing Examination (CNMLE). Similarly, Kasai et al. (2023) evaluate ChatGPT and other GPT family L.LSs on Japanese national medical licensing examinations. They show that GPT-4 passes all six years of the exams, showcas- ing LLMs' impressive capability in language that is typologically distant from English. However, they notice that ChatGPT and GPT3 fail to reach passing criteria and are prone to choosing prohib- ited options.
2020). (4) Claim Prediction: ANS-claim (Khouja.
2020). (5) Machine Generation, for machine- generated text detection (Nagoudi et al., 2020). Paraphrase Detection. The goal of this cluster is to identify the similarity between a pair of sentences from semantic perspective. This cluster contains semantic text similarity (STS) task and paraphrase classification task. For our evaluation, exclude STS, which is regression task and experiment with paraphrase classification using Arabie Q2Q (Seelawi et al., 2019).
Natural Language Inference. This cluster covers the following two tasks: (1) Arabic NLI: Determining whether a text (hypothesis) is false (contradiction), undetermined (neutral), or true (entailment), given a text (premise). This task uses the Arabic part of XNLI corpus (Conneau et al., 2018). (2) Fact-checking: The two datasets Unified-FC (Baly al., 2018) and ANS (Khouja, 2020) are used to target stance and factuality prediction of claims from news and social media. Word Sense Disambiguation (WSD). The Arabic WSD benchmark (El-Razzaz et al., 2021), an MSA context-gloss pair dataset, is used for this task.
Although several works (Alammary, 2022; Abu Farha and Magdy, 2021) benchmark trans- former models on Arabic understanding tasks like sentiment analysis, to the best of our knowledge, our work is the first to evaluate ChatGPT on Arabic NLU and NLG at scale.
B Dataset
B.2 NLG Tasks
B.1 NLU Tasks
We evaluate on four task clusters from ORCA (El- madany al., 2022), as follows:
For NL.G, we create a benchmark using a collection of 23 publicly available datasets from different gen- res. We arrange our NLG datasets into 13 different task clusters:
Sentence Classification. This cluster involves the following tasks and datasets (1) Sentiment Analysis: (Abdul-Mageed et al., 2021b). (2) Social Meaning: Refers to eight social meaning datasets covering prediction of hate and offensive language detection (Mubarak et al., 2020), dangerous speech (Alshehri et al., 2020), sarcasm (Farha and Magdy, 2020), adult content (Mubarak et al., 2021), irony (Ghanem et al., 2019), emotion, age and gender (Mohammad et al., 2018; Abdul-Mageed et al., 2020b). (3) Dialect Identification: Involves three dialect classification levels, binary-level (i.e.. MSA vs. DA), region-level (four regions), and country-level (21 countries). The three tasks are ArSarcasmpia (Farha and Magdy, 2020), AOC dataset (Zaidan and Callison- Burch, 2014), NADI-2020 (Abdul-Mageed et al., 2020a), MADAR (Bouamor et al., 2019), QADI (Abdelali et al., 2020), and Habibi (EI-Haj.
Machine Translation and Dialect Translation. The MT cluster is built around the tasks of X + MSA, where we test the ability of ChatGPT to trans- late from four foreign languages into MSA. For this, we use the United Nations Parallel Corpus (Ziem- ski et al., 2016) that covers the six official UN languages: Arabic, English, French, Russian, and Spanish. The dialectal translation cluster consists of Arabic Dialects + English, where we focus on MT from five Arabic dialects into English using the Multi-dialectal Parallel Corpus (MDPC) proposed by Bouamor al. (2014). MDPC is a human- translated collection of 1K sentences in Egyptian, Tunisian, Jordanian, Palestinian, and Syrian Arabic, in addition to English.
built using six datastes:
Code-Switching. The purpose of the code- switching (CS) task is to translate Arabic dialec- tal text involving code-switching from a foreign language into that foreign language. We use two human-written (natural) code-switched parallel test
"1 From ORCA, we exclude topic classification datusets since these typically have long sequences that can often exceed ChatGPT maximum length of 4, 096 tokens.

---

sets proposed by (Nagoudi et al., 2022b): (1) DZ- FR + FR. consists of code-switched Algerian dialect and French Twitter posts. These posts are manually translated into monolingual French. (2) JO-EN + EN. This is collected from Jorda- nian Twitter and consists of code-switched Jorda- nian dialect and English posts, which are manually translated into monolingual English.
use QALB 2014 (Mohit et al., 2014), a manually corrected collection of Arabic texts from online comments written by native Arabic speakers (L1) in Aljazeera articles. The dataset is divided into a training set with 19.4K sentences, a development set with 1.02K sentences, and a test set with 968 sentences.
Dialogue Generation. We use the open-domain dialogue generation dataset for Arabic dialects proposed by (Naous al., 2023). The dataset consists of 1K pairs of utterances and responses, which were translated from the English DailyDia- log dataset (Li et al., 2017) by three native transla- tors from the Levantine, Egyptian, and Gulf areas. Diacritization. Arabic Text Diacritization (ATD) is the process involving adding missing diacrities words word sequences in Arabic orthography. To accomplish this, we use the Arabic Diacritization dataset proposed by (Fadel et al., 2019).
Summarization and Title Generation. For this cluster, we use XLSum (Hasan et al., 2021), a diverse, multilingual summarization dataset from BBC news supporting 44 languages (including Ara- bie). The Arabic part of XLSum is divided into 37.5K for Train and 4.7K for each of the Dev and Test splits. The news articles in XLSum are anno- tated with summaries and titles, allowing us to use the articles and corresponding titles to evaluate our tille generation models.
Question Answering and Generation. For both of these tasks, we use TyDiQA (Artetxe et al., 2020), a publicly available, multilingual, human-translated QA datasets. For the QA task, we use (Iput:: pas- sage, question, and Output: answer) triplets. For the QG task, switch these (as in Input: passage, answer, and Output: question).
C Anisotropy in BERT Models
A prevalent issue with language models is that they suffer from anisotropy (Ethayarajh, 2019; Li et al., 2020) in the embedding space. That is, representa- tions obtained by the models tend to occupy nar- row cone in the hyperspace, making them less infor- mative. This tasks like WSD because the models need to differ- entiate the representation of the queried word from the representation of the whole sentence. Forming a good representation of the queried word can help the model to align with the given sense during the finetuning. As the representation of both the word and the sentence are very close due to anisotropy. the model cannot properly align the word with the given sense even after finetuning. Therefore, we suspect that anisotropy might potentially be the rea- son behind the inferior performance of MARBERT on WSD task (Table 1).
Transliteration. The goal of transliteration (TS) is to accurately convert word or text from one writ- ing system to another, while maintaining the origi- nal language's pronunciation and sound. For that, we use ANETAC dataset (Ameur et al., 2019), an English-Arabic named entity transliteration dataset. It includes 79, 924 pairs of named entities in En- glish and Arabic, which are categorized into three classes: person, location, or organization.
potentially impact negatively on
Paraphrasing. For this task, we use TaPaCo (Scherrer, 2020) a paraphrase cor- pus that comprises 73 languages, including Arabic. It was extracted from the Tatoeba database and created by aligning sentences with the same meaning. The Arabic portion of TaPaCo, called AraTaPaCo, contains 3K pairs of sentences.
D ChatGPT Exhibits False-Toxied
Text Rewriting, The main objective here is to pro- duce text in the target style while maintaining the content of the original input text. We use the Arabic Parallel Gender Corpus (APGC), which was proposed by Alhafni et al. (2022). This corpus con- tains pairs of sentences where the input sentence is in one gender (e.g., male) and the target sentence has the same meaning but is in the opposite gender (i.e., female).
As Table shows, ChatGPT exhibits significantly poor performance compared to the finetuned AraT5 model on the toxic text classification tasks. Given the concern that models such as ChatGPT might not be aligned well for toxic and harmful language, focus our analysis in part on the toxic language datasets in our evaluation benchmark. To this end, we compute the confusion matrices presented in Figure 7. As we can see, ChatGPT is extremely prone to flagging non-toxic texts as toxic (i.e., a
Grammatical Error Correction, For this task, we

---

abuave
Tall sceach
offeaiive
Thue label
tabet
Thu labol
somal
Prodicted label
Predicted
Predicted
(a) Abusive
(b) Hate Speech
(c) Offensive
Figure 7: Confusion Matrix the toxic text classification tasks.
high rate false positives). We hypothesize that this may be due to one or more of the following reasons:
of any length shall be accepted as long as it sum- marizes the input.
Paraphrasing: Given the input text provided, we ask the model to generate a paraphrase for the in- put.
Lack of diverse Arabic texts in ChatGPT pre- training data (e.g., not enough data from tain dialects).
Machine Translation: Given input in lan- guage, we ask the model translate it into Arabic any other language).
ChatGPT is supervised alleviate the spread of harmful contents, as documented in Ope- nAI safety standards
Text Re-rewriting: Given an input text, we ask the model to rewrite in a specific style. If we are not providing the length/size of the rewriting in the prompt, the output of any length shall be accepted as long as it is accurate without missing/adding information on its Open-ended Dialogue Generation: For each given utterance prompt, the task is generate a reply. The reply is open-ended and is acceptable as long as it is coherent with the prompt.
E Prompt Template for GPT-4 Evaluation
Figure presents the prompt template that rate the model's reponse with GPT-4.
F Class Distribution for Few-shot Examples
We provide class distribution (in percentage) of the classification tasks (except for Dialect-Country where the number of classes is higher than the num- ber of shots) for the full training dataset and our 10-shot sampling in Table and Table 5, respec- tively.
H Examples of Prompt
We present some sample prompts of Arabic NLU and NLG tasks that we use for ChatGPT evaluation in Table 6 and Table 7 respectively.
Examples of Models' Response
As evident from Table 4 and Table 5, the class distribution of the shw-shot samples is reasonably aligned with the corresponding tasks. Therefore, the sampling process of few-shot prompting indeed respects the class distribution of the full training set for the respective tasks the ORCA benchmark.
We present the examples of models' responses the corresponding prompt in Table
Examples of GPT-4 Evaluation
We present the examples of GPT-4 evaluation to the corresponding prompt in Table 9.
Human Evaluation Framework
G.1 Task Description
K
GPT-4 Explanation on Models' Evaluation
Summarization: Given a long text, we ask the model to summarize it. we are not providing the length/size of the summary in the prompt summary
In addition to, rate the models' responses, we gen- erate the rationale behind the ratings by GPT-4. We manually analyze a subset of responses. We find
Phhtpss///ppnna http://pnnii..o//sffetttttttttme

---

You are helptu and precise asisaan for checking the guiliy of reguese. [Propp]]
[Th Sun Assiaut I's Answel
Ausoiezi snswer
TThe Ausiiuuu I's Naswer)
Assiitati stawer
Asdeare Anowrr)
The Star Amiitast 3's Answel! Aodisort
Anower]
[yyorm]
We would like requeet your foalback un the periimmane: thec Al soiannie ropuree the prrngt daplayct above.
Please
based their ouroaaas, intraction felloving capukilit, selevase oakerensy, and fluncyy Eack aviatant revci ovenal raling indietes
of
indicsss best oveall
performance
the asiitarr
you
infiating scoes Astiaat respectivlly scores seportted space ersoquentiie line, piease provide corppehensive ong eeporrmoon of evatoioon avolding any powonal eecring that onder which the reppotes were presensad dlocs noe aflon your julgnient
Please oupur single comsining only
Figure 8: Prompt templates for GPT-4 evaluation. We generate the ratings with GPT-4 for three models (ChatGPT, BL OLMOM and GPT-4) given the input example.
task (full dataset)
class
class
class class
class 4
class 5
class 6
class
binaryy
59.31
machine-gen
49.87
50.13
dialect-region
31.0
0.07
44.17 62.79
24.76
ahusive
29.19
hate speech
5.12
94.88
oftensive
79.95
20.05
irony
48.03
51.97
sarcasm
15.91
84.09
dangcrous
71.16
28.84
adult
11.86
88.14
gender
53.31
46.69
age
33.98
30.26
35.76
claim
67.5
32.5
emotion
13.75
15.13
13.3
2.43
8.44
15.53
sentiment
55.04
31.9
13.06
mq2q
0.01
44.76
55,23
stance
34.05
63.57
2.38
xnli
33.05
33.69
33.26
50.01
Table 4: Class distribution for ORCA classification tasks.
that the ratings of GPT-4 are based generation quality, fluency, accurateness, coherency, and over- all instroction-folloviing capability. We present such examples GPT-4's explanations on the gen- erated ratings to the models' response in Table 10.
L Human Evaluation
We provide the detail human evaluation on the 8 datasets in Table 11

---

task (10-shot)
class
class
class 2
class_3
class 4
class_5 5
class
class 7
dialect binary
60.0
40.0
machine-gen gen
50.0
50.0
dialect-region
40.0
0.0
40.0
20.0
abusive
10.0
20.0
70.0
hate speech
10.0
90.0
offensive
80.0
20.0
irony
40.0
sarcasm
20.0
80.0
dangerous
70.0
30.0
adult
10.0
90.0
gender
60.0
40.0
30.0
40.0
80.0
20.0
emotion
50.0
0,0
10.0
10.0
20.0
0.0
10.0
0.0
60.0
20.0
10.0
mq2q
0.0
30.0
70.0
stance
40.0
50.0
10.0
xnli
40.0
30.0
30.0
wsd
50.0
50.0
Table Class distribution 10-shot sampling.

---

Rating
Criteria
Rating-A
The output is acceptable, valid, and satisfying response to the input prompt. Eg: If we ask the model to summarize an input, the produced output should clearly summarize the input text. For paraphrasing and machine translation, the output is fluent and contains the same semantic meaning as the input.
Output is fluent, relevant, and natural response to conversation.
The produced output correct but at the same standard humans. This is applicable for tasks like open-ended dialogue generation. Irrespective of language vs domain expert, formal or informal), as long as the generated response meets criteria it should be rated as A.
and tone (layman
Rating-B
has followed the task given in the prompt but has a minor issue that needs be improved.
Partially acceptable responses shall be labeled as B.
output has fulfilled the task specified in the prompt with >= 50% accuracy.
The task-specific criteria for 'B' is follows:
Summarization: The model has omitted some key information from the input text in the summary. The model has added some information in the summary that's not in the input text. Minor overlap between summary and text input. Paraphrasing: The output has minor grammar issues. Minor overlap between input text and generated paraphrase.
Open-ended Dialogue Generation: The generated response has minor factual syntactical issues. The generated response has little irrelevant information. Machine Translation: The output has minor fluency issues. The output has minor syntactical issues. The output may contain few irrelevant information.
Rating-C
The output is relevant and the model attempts do the task specified in the prompt but has a major issue in the quality of the output.
The produced output is <50% accurate for the task specified in the prompt.
The task-specific criteria for 'C' is follows:
Summarization: The model has omitted significant information from the input text in the summary. The model has added significant information in the summary that's not in the input text. Major overlap between summary and text input. Paraphrasing: The output has major grammar issues. The output is the same as input text with a few minor word replacements. Major overlap between input text and generated paraphrase.
The output has a major semantic meaning difference compared to the input. Open-ended Dialogue Generation: The generated response has major factual or syntactical issues. Generated response is machine-like, non-fluent, and/or irrelevant.
Machine Translation:
The produced translation may contain one or two dif- ferent language tokens except for the target language. The output has major grammatical, syntactical, and fluency issues. The semantic meaning is a little different than the input.
Rating-D
The output invalid and totally unacceptable for the task specified in the prompt.
The produced output is not at relevant to the task specified the prompt. Eg:
The model didn't 't produce any output at all.
The model has refused to provide answer. Eg: "As Al language model I do not have sufficient...
The model just copies the input as the output
The produced output is in a different language.

---

Task
Shot
Prompt
Abusive
0
You are an annotatorGPT. Your task is do abusive language classification. You will be given an input text and you should predict the label of the text from the list: ['hate', 'normal', "abusive"]. Now predict the label the following input:
Input: صراي Iamdt + ميو Output:
Dialect Binary
You are an annotatorGPT.. Your task is to do dialect classification between Dialectal Arabic (DA) vs Modern Standard Arabic (MSA). You will be given an
input text and you should predict the label of the text from the list: ['DA', 'MSA'].
Now predict the label of the following input: Inpution حسن بيي التادي مرو الف
Output:
Paraphrase
You are an annotatorGPT. Your task is do duplicate/paraphrase detection. You will be given two input texts and you should predict the label from
the list: ('duplicates', 'not duplicates") J.. Now predict the label of the following input:
Input:
Text 1: f p. العطمم الصين سور طول بلع &
Text العطلي الصين لسور المدردية الإمتعادات هي lo
Output:
Sarcasm
You are an annotatorGPT. Your task to sareasm text classification. You will be given an input text and you should predict the label of the text from the list: ['sarcastic', 'non-sarcastic'].
Now predict the label of the following input:
Input: SALe فيقي عار بيقرججي لي المدع اسفل معا مناك
Output:
Sentiment
3
You an annotatorGPT. Your task is do sentiment classification. You will be given an input text and you should predict the label of the text from the list: ['positive', 'negative', 'neutral'). Here, we provide you examples of the task: Example:
Input: de; راحعاا
Output: negative
Example: 2
Input: الأرمم الأاار عن ببحنون ساققون لسقق ميدانية جولة
Output: neutral
Example: 3
Input: USER ماللمل بسوون أمدامم كفف الل ماعاء
تقسع عي 1ةتتو مfع مررم الرياف امانة مككا بااقكرة الططوي
Output: negative
Now predict the label the following input:
Input: غجزتوا all nhtte وأعر
Output:
Table Prompt examples for some Arabie NLU tasks.

---

Task
Shot
Prompt
Question Gen
You generative language model. Your task is to do question generation. You will be given context and the answer of the question imput. You should generate the question based on the context and the answer. The output must be in Arabic. You do not need to provide explanation.
Now generate the output of the following input: Input:
Context:
T للوج فراا إع اتقلتت & خيااا في woo عام واتت
Ge انروابتت ماري توحت ملكة lبييا هريا الأكة اباء أصم رمي
ف 3o jky laje من ععر الربمة في رحي عشر السادس لريس اللك
of من عهو اmاttسة
Answerligg
Output:
Diacritization
You generative language model. Your task is do text diacritization. You will be given input Arabic text and you should generate the diacritized version of the output. The output must be in Arabic. You do need provide any explanation. Now generate the output of the following input:
Input: والععياا الشسل جوار مدل أي C العيد R وولء
Output:
MT (en (en-var)
You generative language model. You task is do machine translation. You given English text the input, you should translate Arabic. You do not need provide any explanation. Now generate the output of the following input: Input: It publishes handhook offering selection of first-class academic programmes and providing key information to help understand the French academic system. Output:
CST (Jo-en-ven)
You generative language model. You task is do code switched machine translation. You given Jordan Arabic and English mixed text the input, you should translate to English. You do not need provide any explanation. Now generate the output the following input: Input:
ببm بست 43g مافف ب Aell 4ييا lay Rلقا!ة
Output:
Dialogue (EGY)
3 You are generative language model. You task to do dialogue generation. You are given an imput text, you should generate an output in reply to that input text. The output must be in the same Arabic dialect as input. You do not need to provide any explanation. Here, we provide you examples of the task:
Example: 1
Input: امقف حساار
Output: جااlة Cutb أكون عفان ee ,5 say
Example: 2
Input: Las خدو انو توطك عe sel
Output: اووي Assiiii
Example: 3
Input: الأكمن بنب
Output: المياا اللا Zامة ol Now generate output the following input: الزور ملحلل معن رلي
Input:
Output:
Table 7: Prompt examples for some Arabic NLG tasks.

---

Prompt
Gold
BLOONZ
ChatGPT
GPT-4
You are a generutive language model. You task is to do ma- chine trunslation, You are given a French text as the input, you
ااو الرهق
العي الرفق
الااي اللحق
الااي اللحق
should translate it to Arabie, not need to provide any explanation. Now generate outpur following inpur:
Input: amnexe ii Output:
are generative language model. You task logue generation. You are given an input text, you should gen- erate an output in reply that inpul
dia-
4 التور Cluo كيي مio feseel wlll
النور مhu fلeil كنته واسرورة
. 4ا الور cho
outpul must
same Arabic dialect as in- provide
put. You not any explanation.
Now generate output following input:
Input: .uPel 4 الرر صيعع
Output:
generative language model. You task do dia- logue generation. You given inpul texl, you should gen- erate output in reply that input The outpul must be the same Arabic dialect as in- You do not need to provide any explanation.
معن مررف
LA بررقنن في من ففتبا
ألوي pi سي دaي
f sس is
East ويع
Now generate the output of the following input:
Input:
از مدديي سوe
Outpat:
generative language model. You task do code switched machine translation. You given Jordan Arabic and English mixed input. you should translate to English. You do not need to pro- vide any explanation.
والزااي الأمل JL Ragao fake acc
Family and relatives are more calumity than fake acc
Family and relatives are bigger problem than fake accounts.
Family and relatives are more trouble than accounts.
the
Now generate the ourput of the following input:
واقام ااهل
JI 3 157 مميية
ace
Output:
are generative language model. You task is vode
switched machine translation. are given a Algerian Ara- bie and Freach mixed input, you should translate to French. You vide any explanation.
Bonjour les amies
Bonjour mes amis
Bonjour mes chers.
Bonjour mes hicn-aimés
need to pro-
Now generate the ourput following input:
Input: Bonjour اي ا
Output:
Table 8: Examples of models' responses.

---

Prompt
GPT-4 Rating
You are a helpful and precise assistant for checking the quality the response.
(Prompt)
You are generative language model. You task is to do machine translation. You are given a French text as the input, you should translate it to Arabic. You do not need to provide any explanation. Now generate the output of the following input:
Input: I' e'eiil a continué d' exécuter des civils.
Output:
(The Start of Assistant I's Answer)
ADA
الننون all إعاا! الملااية أللراة تططي واميل
(The End of Assistant I's Answer)
(The Start of Assistant 2's Answer)
المنت تفية ف اسنرر
(Thee End of Assistant 2's Answer)
(The Start of Assistant 3's Answer)
المنوان أماام ف داك تطاي اسرر
(The End of Assistant 3's Answer)
(System)
We would like to request your feedback on the performance of three Al assistants in response to the prompt displayed above.
Please rate the responses based their correctness, instruction-followin capability, relevance, co- herency, and fluency. Each assistant receives an overall rating on scale of 'A' to 'D', where 'A' indicates the best overall performance and 'D' indicates the worst overall performance. For example, if the Al assistant just copy/paste the input text, you should rate it as 'D'.
Please first output a single line containing only three values indicating the scores for Assistant 1, 2. and respectively. The three scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
You are helpful and precise assistant for checking the quality the response. (Prompt) You generative language model. You task is do code switched machine translation. You are given a Jordan Arabic and English mixed text as the input, you should translate it to English. You do not need to provide any explanation. Now generate the output of the following input: Input: Yess and my dkklsssillss are getting much better now!
Output:
(The Start of Assistant I's Answer)
ADB
yes and my plumbing skills are getting much better now!
TThe End of Assistant I's Answer)
(The Start of Assistant 2's Answer)
yess and my english skills are getting much better now!
(The End of Assistant 2's Answer) (The Start of Assistant 3's Answer) yess and my threading skills are getting much hetter now!
(The End of Assistant 3's Answer)
[System]
We would like to request your feedback on the performance of three Al assistants in response to the prompt displayed above.
Please rate the responses based their correctness, instruction-followee following capability, relevance, co- herency, and fluency. Each assistant receives an overall rating on scale of 'A' to 'D', where 'A' indicates the best overall performance and 'D' indicates the worst overall performance. For example, if the Al assistant just copy/paste the input text, you should rate it 'D'.
Please first output a single line containing only three values indicating the scores for Assistant 1, 2, and respectively. The three scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
Table 9: Examples of GPT-4 ratings on the models' responses.

---

Task
Input
Model Response
GPT-4 Rating
GPT-4 Explanation
Assistant changed the num- ber brothers in the para- phrase "alis أمعر t "لبل"" (You have younger brother), which does not preserve the original semantic meaning the input text. Hence, Assis- tant 2 receives a 'C' rating. did attempt paraphrase, but failed to retain key informa- tion from the original.
أحون لبيهه منلك أصخر
टी للبلله منiك أصرر
Paraphrasing
C
Assistant 3's translation, on the other hand, seems to be misspelled and grammatically incorrect. It seems that there is a typing error the re- sponse, which causes the sen- tence to not make sense Arabic. Hence, Assistant 3 is given the grade 'D'.
MT (en-var)
autres dépenses تومررر de personnel
D
الأحري طلفين
Assistant 2, however, incor- rectly translated the Arabic text G "زو (which means "you should marry" in femi- nine form) into "se marier" (to marry general) French. This translation is not accurate because changes the mean- ing and the recipient of the original sentence. Therefore, Assistant 2 gets the grade 'C',
CST (Dz-fr-frr)
11 faut تلويي
se marier
Assistant I's response is brief and litte ambiguous, but it's relevant and maintains the same dialect. While it tech- nically answers the question, it lacks some depth and detail which would have been help- ful. Therefore, I give it 'B' rating.
Dialogue (EGY)
تلهي تصرف go?
ايب كيم بغك مف
B
Assistant correctly translates the Arabic part of the input text into English, maintaining its original context and mean- ing. The response is coherent, relevant, and follows the given instructions correctly, earning Assistant 'A' grade.
miss you singing "lego house" to me.
CST (Jo-en-een)
تمنيل المتتت lego house
Table 10: Examples of GPT-4's explanation the evaluation of the generated responses.

---

ChatGPT
BLOOMZ
GPT-4
Task
Annotator
A
B
D
A
B
C
D
A
B
C
D
1
42
6
2
2
2
I
45
43
0
7
Paraphrasing
2
45
3
2
4
0
2
44
41
7
2
0
Avg
43,5
4.5
2
3
1
1,5
44.5
42
3,5
1
43,5
Sum
87
9
4
6
2
3
89
84
2
7
43
4
4
0
22
12
5
11
43
0
0
Summarization
2
42
6
0
4
0
30
16
48
0
0
Avg
42,5
5
0
13
17.5
13.5
45.5
4.5
0
0
Sum
10
26
35
26
14
9
2
5
4
39
43
1
0
CST (Dz-fr-fft)
2
12
3
45
Avg
27.5
13
7.5
2.5
4.5
44
0.5
Sum
55
26
13
5
9
8
78
88
11
1
0
3
3
5
38
17
37
CST (Jo-en-ven)
Avg
29
13
6.5
1.5
4.5
4.5
36
37.5
2.5
Sum
58
26
13
3
9
9
10
72
75
18
5
2
14
15
8
22
2
38
11
4444
15
24
41
(Fr-AAr)
Avg
37
12.5
0.5
15
7.5
23,5
31.5
18
0.5
0
Sum
74
25
0
30
15
47
63
36
0
1
26
17
O
0
50
23
14
4
9
2
15
7
23
5
0
50
19
17
4
10
Dialogue (EGY)
0
Avg
20.5
12
14.5
0
50
21
15.5
4
9.5
Sum
41
24
29
0
100
42
31
8
19
9
30
3
0
0
49
25
16
8
2
0
0
0
50
40
2
5
Dialogue (LEV)
Avg
11.5
24.5
0.5
49.5
32.5
9.5
5
Sum
23
16
65
6
4
27
1
0
0
0
25
14
2
31
28
Dialogue (GUL)
Avg
4.5
29
3,5
0
0
16
21
2
Sum
26
9
58
7
0
0
0
100
32
22
42
4
Table 11: Human evaluation results.


Reviews:
[{"review_id": "jwSy1LuxMR", "reviewer": "Reviewer_ikv4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful review. We are glad that you find our paper has addressed a significant research gap by investigating ChatGPT’s performance on Arabic NLP. Please find our response to your comments below.\n\n**W1. The paper mentions that there is limited research on evaluating ChatGPT's performance on languages other than English, but it does not sufficiently integrate or discuss existing studies on NLP in non-English languages.**\n\nConsidering the number of languages ChatGPT has not been evaluated on at the time of submission, we believe that our claim that there is limited research on non-english remains true. The research gap is even more severe when it comes to languages with a wide range of dialects such as Arabic. In our paper, we cite a total of ten works related to evaluation of LLMs such as ChatGPT and BLOOMZ on non-English, as follows:\n\nFirst, we cover six notable works that evaluate ChatGPT on multilingual tasks (please see “Performance on Multilingual Tasks”, Section 2, lines 143-172). For example, we cite [1, 2] in which the authors evaluate ChatGPT/GPT-4 on medical licensing exams in Chinese and Japanese, respectively. Additionally, we include four more references particularly related to machine translation tasks involving non-English in Appendix A.1. \n\nThat being said, since our work focuses on evaluating ChatGPT and other LLMs on Arabic, we also cite a number of previous work on Arabic NLP. This includes, for example, sources of our downstream datasets. \n\nRegardless, even though our paper has more than four pages of references, there is a chance that we may have missed some important relevant work and we will extend our literature review in the camera ready. For example, we will cover works that evaluate smaller models dedicated to Arabic.\n\n\n**W2. While the paper compares ChatGPT's performance with smaller fine-tuned models, it could benefit from more in-depth analyses of the differences in capabilities between the models. For example, specific tasks where ChatGPT falls short compared to dedicated models could be discussed in greater detail.** \n\nWe believe this is a good suggestion and we will enhance our analysis about the differences in capabilities between the models. Meanwhile, in the analysis we provided we have offered several types of meaningful comparisons including on the differences between models. We summarize these here:\n\nWe find that ChatGPT (and BLOOMZ) falls short compared to smaller finetuned models on the majority of the NLU and the NLG tasks (30 out of 35 tasks; Table 1 and Table 2), revealing the weakness of ChatGPT (and BLOOMZ) on Arabic (lines 106-109). We offer an analysis of this finding across the different tasks we consider. For example, we thoroughly discuss the limitation of ChatGPT on some sensitive tasks like toxicity detection (Appendix D), showing that ChatGPT exhibits higher false-toxicity rate. We further analyze tasks where ChatGPT indeed outperforms the dedicated models. For example, for WSD task, we suspect the reason is anisotropy (lines 352-355) and discuss this in detail in Appendix C. For code-switched translation tasks where ChatGPT also outperforms the smaller finetuned model, we point out that ChatGPT was likely exposed to other high resource languages like French during the pretraining and hence can perform well on code-switching. \n\nIn addition, we provide a comparative analysis between the large language models and the smaller finetuned models across different Arabic varieties. For example, we show that limitations of the larger models such as ChatGPT and GPT-4 are more acute for Arabic dialects (rather than MSA) (lines 487-490). This can be directly correlated to availability of dialectal  pretraining data (lines 501-505) that were included in these LLMs. Since the smaller models are computationally less costly, we show that they can be easily finetuned on dialectal data to acquire high performance. \n\nAbove all, we would also like to emphasize that it is difficult to provide a conclusive analysis of the outcomes of closed models such as ChatGPT due to lack of knowledge as to how these models are trained and what datasets have been used during their pretraining.\n\n\n**W3. The paper states, \"We randomly sample from each respective training dataset.\" Does this sampling process consider the class distribution for certain tasks? I believe further clarification is needed regarding the experimental setups. Additionally, the authors assert that MARBERT achieves the best results in the literature, but they have not provided sufficient references to support this claim.**\n\nWe thank the reviewer for this comment. We are providing the class distribution (in percentage) of the classification tasks (except for Dialect-Country where the number of classes is higher than the number of shots).\n\n| task (full dataset) | class_0 | class_1 | class_2 | class_3 | class_4 | class_5 | class_6 | class_7 |\n|--|--|--|--|--|--|--|--|--|\n| dialect-binary | 59.31 | 40.69 |  |  |  |  |  |  |\n| machine-gen | 49.87 | 50.13 |  |  |  |  |  |  |\n| dialect-region | 31.0 | 0.07 | 44.17 | 24.76 |  |  |  |  |\n| abusive | 8.02 | 29.19 | 62.79 |  |  |  |  |  |\n| hate speech | 5.12 | 94.88 |  |  |  |  |  |  |\n| offensive | 79.95 | 20.05 |  |  |  |  |  |  |\n| irony | 48.03 | 51.97 |  |  |  |  |  |  |\n| sarcasm | 15.91 | 84.09 |  |  |  |  |  |  |\n| dangerous | 71.16 | 28.84 |  |  |  |  |  |  |\n| adult | 11.86 | 88.14 |  |  |  |  |  |  |\n| gender | 53.31 | 46.69 |  |  |  |  |  |  |\n| age | 33.98 | 30.26 | 35.76 |  |  |  |  |  |\n| claim | 67.5 | 32.5 |  |  |  |  |  |  |\n| emotion | 29.62 | 1.8 | 13.75 | 15.13 | 13.3 | 2.43 | 8.44 | 15.53 |\n| sentiment | 55.04 | 31.9 | 13.06 |  |  |  |  |  |\n| mq2q | 0.01 | 44.76 | 55.23 |  |  |  |  |  |\n| stance | 34.05 | 63.57 | 2.38 |  |  |  |  |  |\n| xnli | 33.05 | 33.69 | 33.26 |  |  |  |  |  |\n| wsd | 49.99 | 50.01 |  |  |  |  |  |  |\n\nWe are further providing the class distribution of the 10-shot training samples that we randomly selected.\n\n| task (10-shot) | class_0 | class_1 | class_2 | class_3 | class_4 | class_5 | class_6 | class_7 |\n|--|--|--|--|--|--|--|--|--|\n| dialect-binary | 60.0 | 40.0 |  |  |  |  |  |  |\n| machine-gen | 50.0 | 50.0 |  |  |  |  |  |  |\n| dialect-region | 40.0 | 0.0 | 40.0 | 20.0 |  |  |  |  |\n| abusive | 10.0 | 20.0 | 70.0 |  |  |  |  |  |\n| hate speech | 10.0 | 90.0 |  |  |  |  |  |  |\n| offensive | 80.0 | 20.0 |  |  |  |  |  |  |\n| irony | 40.0 | 60.0 |  |  |  |  |  |  |\n| sarcasm | 20.0 | 80.0 |  |  |  |  |  |  |\n| dangerous | 70.0 | 30.0 |  |  |  |  |  |  |\n| adult | 10.0 | 90.0 |  |  |  |  |  |  |\n| gender | 60.0 | 40.0 |  |  |  |  |  |  |\n| age | 40.0 | 30.0 | 40.0 |  |  |  |  |  |\n| claim | 80.0 | 20.0 |  |  |  |  |  |  |\n| emotion | 50.0 | 0.0 | 10.0 | 10.0 | 20.0 | 0.0 | 10.0 | 0.0 |\n| sentiment | 60.0 | 20.0 | 10.0 |  |  |  |  |  |\n| mq2q | 0.0 | 30.0 | 70.0 |  |  |  |  |  |\n| stance | 40.0 | 50.0 | 10.0 |  |  |  |  |  |\n| xnli | 40.0 | 30.0 | 30.0 |  |  |  |  |  |\n| wsd | 50.0 | 50.0 |  |  |  |  |  |  |\n\n\nAs evident from the tables, the class distribution of the few-shot samples is reasonably aligned with the corresponding tasks. Therefore, the sampling process indeed considers the class distribution of the respective tasks. We will add this in the appendix of the paper.\n\nRegarding the performance of MARBERT, we referred to the original paper (line 248), where the authors claim the superiority of the model. We will put the same reference again along with line 251.\n\n\n**W4. The paper appears quite condensed. As a result, I would prefer a more comprehensive evaluation focused on a few tasks, delving deeply (evaluating ChatGPT and making comparative analysis with fine-tuned models for Arabic), rather than a broad approach across 44 tasks with concise explanations.**\n\nWhile we agree to the benefit of focusing on fewer tasks, we also see the privilege of providing a comprehensive evaluation across a large number of tasks that represent Arabic at scale. \n\nThat being said, we also methodically group tasks into either natural language understanding (NLU) or natural language generation (NLG). This breakdown immediately demonstrates capabilities of the models across these two broad categories. We also provide analysis on some selected tasks like toxicity detection (false toxicity in Appendix D), WSD (anisotropy in Appendix C). One challenge we have is page limits, and we wanted to strike a balance between what we put in the main body of the paper and a reasonable length for appendixes. However, with an extra page for the camera-ready, we will provide a more in-depth analysis of ChatGPT on selected tasks following the suggestion. For example, we have further probed ChatGPT’s code-switching ability using diagnostic test cases. Specifically, we manually evaluated ChatGPT’s capability of English mixed MSA, Egyptian, and Moroccan code-switched translation generation from plain English text. We ask two annotators who are fluent on the respective varieties (as well as English), to evaluate a the diagnostic dataset based on fluency (defined as how fluent the translated text is), faithfulness (defined as how semantically close the translated text is to the source text), and code-switching ability (defined as how accurately the translated text includes the code-switch scripts). We provide clear instructions to annotators about the task. We present our result (in percentage) in the table below, and will provide the details of the full analysis in the camera-ready.\n\n\n| Annotator | CST | fluency (A/B/C/D) | faithfulness (A/B/C/D) | code-switching (A/B/C/D) |\n|--|--|--|--|--|\n| Annotator 1 | En->MSA_En | 80/10/10/0 | 80/20/0/0 | 0/40/0/60 |\n|  | En->Egy_En | 70/10/20/0 | 100/0/0/0 | 20/30/0/50 |\n|  | En->Mor_En | 10/40/50/0 | 90/0/10/0 | 20/30/0/50 |\n|  | ***Avg*** | 53.3/20/26.7/0 | 90/6.7/3.3/0 | 13.3/33.3/0/53.4 |\n| Annotator 2 | En->MSA_En | 30/70/0/0 | 90/0/10/0 | 0/30/0/70 |\n|  | En->Egy_En | 50/40/10/0 | 90/10/0/0 | 0/40/0/60 |\n|  | En->Mor_En | 20/40/40/0 | 100/0/0/0 | 20/30/0/50 |\n|  | ***Avg*** | 33.3/50/16.7 | 93.4/3.3/3.3/0 | 6.7/33.3/0/60 |\n| | ***Average*** | 43.3/35/21.7/0 | 91.7/5/3.3/0 | 10/33.3/0/56.7 |\n\n\n\nWe find that ChatGPT produces fluent translations that are also semantically close (faithful) to the source text. However, ChatGPT struggles to produce code-switched text and generates mostly in Arabic script. Interestingly, this issue is more prevalent for MSA than Egyptian and Moroccan. We hypothesize that the tokens in the source English text are very common in MSA. As a result, the model does not seem ‘willing’ to code-switch the words into English.\n\n**W5. The paper lacks a discussion on the interpretability of ChatGPT's outputs. Providing insights into why the model may perform differently on Arabic and its dialects, for example by highlighting challenging Arabic features, could enhance the paper's depth.**\n\n\nWe have carried out an analysis of ChatGPT’s ability to translate from different varieties of Arabic into English. In this analysis, we observe a number of challenges that we believe are unique to Arabic. This involves a discrepancy between the model’s ability to translate from MSA and their ability to translate from Arabic dialects such as Egyptian or Moroccan. This discrepancy cuts across several linguistic categories and involves a discussion of topics such as translation of endearment expressions; multi-word expressions, idioms, and proverbs; negation; sub-token-level code-switching; and dialectal variants of MSA lexica. Across all of these areas, we observe challenges faced by ChatGPT and we will include this analysis in our camera-ready version of the paper. We believe this analysis will offer further insights and additional depth as to the interpretability of ChatGPT's outputs vis-a-vis Arabic and its dialects.\n\nIn addition to this, we would like to emphasize that our work provides an extensive systematic evaluation (line 67, line 106) that shows how ChatGPT fares on Arabic. Our work demonstrates that ChatGPT performs better on MSA than DA data (Section 8), because the model has likely seen more MSA data (lines 487-489). We further probe the performance variation across the different dialects and conclude that the potential reason is due to the unavailability of dialectal pretraining data (lines 501-505).\n\nMoreover, we conduct human evaluation for those tasks where automatic metrics fail to capture overall quality of the output (Section 9). Our evaluation framework to an extent can be used to interpret output of ChatGPT and other LLMs. For example, we state that rating-A is when the output is acceptable without any issue. While rating-B and rating-C are when the outputs have minor to major hallucination, toxicity, grammatical issues. Rating-D is when the output is not a response to the prompt. Again, we will augment these already offered insights with our new analysis we describe above.\n\n\n**Missing References**\n\n*M1. BERT Models for Arabic Text Classification: A Systematic Review (Ali Saleh Alammary, 2022)*\n\n*M2. Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection (Ibrahim Abu Farha and Walid Magdy, 2021).*\n\n*M3. Many more references in literature for evaluating LLMs models for Arabic tasks and others for evaluating ChatGPT on non-English languages.*\n\nWe thank the reviewer for providing these references. We will add the mentioned references and other relevant literature to our paper.\n\n\n***References***\n\n[1] Wu, Jiageng, et al. \"Qualifying Chinese Medical Licensing Examination with Knowledge Enhanced Generative Pre-training Model.\" arXiv preprint arXiv:2305.10163 (2023).\n\n[2]. Kasai, Jungo, et al. \"Evaluating gpt-4 and chatgpt on japanese medical licensing examinations.\" arXiv preprint arXiv:2303.18027 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- The paper addresses a significant gap in NLP research by investigating the performance of ChatGPT on Arabic and its dialects.\n\n2- The paper goes beyond simple evaluation and includes a comparative analysis with smaller, fine-tuned Arabic models. This comparison helps highlight the specific strengths and weaknesses of ChatGPT in handling Arabic tasks.\n\n3- The paper establishes a benchmark for evaluating ChatGPT's performance on Arabic tasks, which can serve as a reference for future studies.", "reasons_to_reject": "1- The paper mentions that there is limited research on evaluating ChatGPT's performance on languages other than English, but it does not sufficiently integrate or discuss existing studies on NLP in non-English languages.\n\n2- While the paper compares ChatGPT's performance with smaller fine-tuned models, it could benefit from more in-depth analyses of the differences in capabilities between the models. For example, specific tasks where ChatGPT falls short compared to dedicated models could be discussed in greater detail.\n\n3- The paper states, \"We randomly sample from each respective training dataset.\" Does this sampling process consider the class distribution for certain tasks? I believe further clarification is needed regarding the experimental setups. Additionally, the authors assert that MARBERT achieves the best results in the literature, but they have not provided sufficient references to support this claim. \n\n4- The paper appears quite condensed. As a result, I would prefer a more comprehensive evaluation focused on a few tasks, delving deeply (evaluating ChatGPT and making comparative analysis with fine-tuned models for Arabic), rather than a broad approach across 44 tasks with concise explanations.\n\n5- The paper lacks a discussion on the interpretability of ChatGPT's outputs. Providing insights into why the model may perform differently on Arabic and its dialects, for example by highlighting challenging Arabic features, could enhance the paper's depth.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study aims to assess ChatGPT's capabilities in Arabic and its dialects. The research involves a comprehensive evaluation, including both automated and human assessments, across a broad spectrum of language understanding and generation tasks (44 tasks) on diverse datasets (60 datasets).\n\nThe focus is on comparing ChatGPT's performance in Arabic with smaller models that have been fine-tuned specifically for Arabic. The findings reveal that, despite its success in English, ChatGPT consistently falls short when compared to these monolingual fine-tuned models. A detailed comparison between ChatGPT and GPT-4 in Modern Standard Arabic and Dialectal Arabic further highlights limitations in handling Arabic dialects.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- BERT Models for Arabic Text Classification: A Systematic Review (Ali Saleh Alammary, 2022)\n- Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection (Ibrahim Abu Farha and Walid Magdy, 2021).\n\nMany more references in literature for evaluating LLMs models for Arabic tasks and others for evaluating ChatGPT on non-English languages."}}, {"review_id": "b3aCoJhA30", "reviewer": "Reviewer_YeJa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your helpful comments. We are pleased to know that you liked the thorough experimental evaluation of this work. Please find our response to your concerns below:\n\n\n**W1. Reading through the description of the varying tasks and their results was a bit tedious since there are so many tasks to cover.**\n\nThank you for your comment. Our intention is to offer a comprehensive evaluation on a broad range of Arabic NLP tasks. We needed to include sufficient description of various tasks and the results we acquired. We believe this is necessary for replication and facilitation of meaningful comparisons that may be conducted in the future. However, we had actually put a very concise description of the tasks in Section 3 and moved the detailed description in Appendix B. \n\n**Typos Grammar Style And Presentation Improvements**\n\n*T1. NLI is used in line 160 for the first time without expanding the acronym.*\n\nWe thank the reviewer for pointing this out. We will spell out “natural language inference” before we introduce the acronym.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Understanding how ChatGPT is performing on varying languages is crucial in understanding how to improve LLMs output on new languages. The paper presents a large number of experiments for various NLP tasks that cover a wide range of real-world problems. ", "reasons_to_reject": "Reading through the description of the varying tasks and their results was a bit tedious since there are so many tasks to cover. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a set of experiments evaluating the performance of ChatGPT on varying Arabic NLP tasks, in both Standard and Dialectal Arabic. The experiments are run on ChatGPT and GPT-4, as well as on other LLMs. The paper also provides a human evaluation to compare the results of varying systems.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "NLI is used in line 160 for the first time without expanding the acronym. "}, "extra_text": {}}, {"review_id": "EfZyz5Z0E3", "reviewer": "Reviewer_Utir", "summary": "", "questions": "A) In Section 4, you mention that an array of prompt templates was evaluated, but English prompts were found to be best. How were candidate prompts evaluated? With what dataset? On all models?\n\nB) Do you constrain the outputs of ChatGPT or BloomZ using logit bias or using logprobs rather than free-form generation? If not, how are responses matched to the label set from a free-form generation?\n\nC) You mention that your annotators are native Arabic speakers, but don't mention their familiarity with dialects. Which dialects are they familiar with and what is their level of familiarity for each dialect? Are they familiar with the non-Arabic languages in the codeswitched evaluations and if so what is their level of familiarity with them?\n\nD) How was inter-annotator reliability measured for each task evaluated with humans?\n\nE) How were the finetuned models trained? Hyperparameters and training data for those models should be reported to be able to reproduce the finetuned baselines used in this work and to get a sense for how reasonable their training process was.\n\nG) Why was AraT5 omitted from the human evaluation? This seems a significant omission since it outperforms the other models by the automatic baselines.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough review. We are encouraged to know you see our paper as helping more practically-oriented researchers working on Arabic NLP. We provide detailed answers to your comments next.\n\n**W1. The authors utilize a static prompt template for all tasks - but LLM prompting can result in extremely high variance and usually needs to be tuned between tasks. Without evaluating multiple prompts per task, these results may be misleading as readers have no sense what the variance is. This only compounds with the reproducibility challenges inherent to evaluations based on changing API endpoints acknowledged in the limitations. Especially since the authors draw conclusions primarily by the relative rankings of models, this work seems to have reasonable risk of some conclusions being misleading/incorrect given different prompt phrasing or as the API changes.**\n\nIn our work, we hypothesize that a good model should respond successfully to a standardized and informative prompt. We also believe that a fixed prompt, so long as it is sufficiently clear, is necessary for a fair comparison between the different models. Without this type of fixed prompt template, it is difficult to evaluate and rank different models across different tasks. For these reasons, we provided the models with a ***standard*** prompt that has also been employed previously for ChatGPT benchmarking on English tasks [1] (Table 27) and multilingual tasks [2].  Our prompt is also ***informative***: we designed a well-articulated prompt defining the roles, the tasks, and the associated expected responses for each specific task. \n\nIn addition, while we agree with the reviewer that variation in prompt can at times change the performance, we do not see the pattern in our result changing in any significant way, especially given the large margin of difference (on average 17.57% for NLU and 4.46% for NLG) between the larger models and the smaller finetuned models. \n\nRegarding your point to the challenges of reproducibility associated with API changes, we believe this is external to our work. However, we have stated the exact API version we deployed (line 238) to facilitate future comparisons to the extent this is possible. Please also note that, similar to other works on ChatGPT evaluation [1][2], we do not claim our benchmarking to be universal across all variations of experimental settings. Rather, we emphasize that our evaluation and benchmarking are reproducible by using the detailed standard prompt we opt for and the exact version of the API that we employ. The impact of the variation of prompting and API versions can be considered as an incremental line of future research.\n\n\n\n**W2. The work's analysis focuses on high level comparison of the performance metrics and postulations at their causes. This means that the primary takeaway from the work is that these models have significant room to grow for Arabic NLP, but they don't offer a deeper argument for how these findings should change existing research directions or what is unique about the challenges for Arabic NLP for LLMs vs. other languages.**\n\nThis is useful feedback and we will include a dedicated section focusing on offering a deeper argument for how our findings can change existing research directions. The reviewer is also right that in our work, (1) we show a significant room for improvement for LLMs like ChatGPT on Arabic (lines 114-119). \n\nMeanwhile, our work offers a number of other important takeaways that can guide future research. For example, since Arabic is a collection of languages and language varieties, (2) we clearly show that LLMs do not handle tasks involving the different varieties and dialects of Arabic equivalently. For this reason, we suggest that future research should focus more on Arabic dialects. This includes involving dialectal pretraining data in these models as well as possibly finetuning models on dialectal tasks (lines 106-109; lines 487-489; lines 501-505). We also delve deeper into this analysis, offering a (3) dedicated section showing the unique challenge associated with Arabic NLP based on the fact that the performance differs between modern standard Arabic and dialectal Arabic (Section 8). To the best of our knowledge, this is the first work to study the comparison of ChatGPT and GPT-4 on Arabic dialectal variation (lines 82-85). (4) We also emphasize the role of GPT-4 as a potential alternative of human evaluators for Arabic language (Section 10). \n\n**Questions For The Authors:**\n\n**QA. In Section 4, you mention that an array of prompt templates was evaluated, but English prompts were found to be best. How were candidate prompts evaluated? With what dataset? On all models?**\n\nWe thank the reviewer for this question. We evaluated Arabic and English prompt templates on both NLU and NLG tasks with ChatGPT and BLOOMZ. Specifically, we evaluated on dialect identification, machine-generated text detection, and toxicity detection for NLU and machine translation tasks for NLG. We evaluate the models on these tasks with both English and Arabic prompts, observing that the English prompt outperforms its Arabic counterpart. This finding is also substantiated by prior literature [2] that observed the superiority of the English prompt even on multilingual tasks (lines 146-150). We will add this explanation in the camera-ready version.\n\n**QB.  Do you constrain the outputs of ChatGPT or BloomZ using logit bias or using logprobs rather than free-form generation? If not, how are responses matched to the label set from a free-form generation?**\n\nWe thank the reviewer for raising this point. We did not use logit bias or logprobs. Rather, we used free-form generation for both ChatGPT and BloomZ. This is because constraining the models to specific output would undermine the instruction-following capability of the models. Hence, we follow [3] to directly predict the answer with free-form generation along with simple post-processing like removing leading or trailing whitespace, and then apply exact matching with the label set. We will add this response in the experimental setting of our paper.\n\n**QC. You mention that your annotators are native Arabic speakers, but don't mention their familiarity with dialects. Which dialects are they familiar with and what is their level of familiarity for each dialect? Are they familiar with the non-Arabic languages in the code-switched evaluations and if so what is their level of familiarity with them?**\n\nThe annotators who contributed to this work are native speakers of the respective dialects they evaluated. The native speakers who evaluated Algerian-French to French translation are native Algerian speakers who have a professional level of expertise in French. The same applies to other code-switching tasks.\n\n**QD. How was inter-annotator reliability measured for each task evaluated with humans?**\n\nWe thank the reviewer for this question. We used Cohen’s kappa metric to measure the inter-annotator reliability. We find the inter-annotator agreement more than 0.6 for the majority of the tasks. We will add the details of inter-annotator agreement to Section 9 of the paper.\n\n**QE. How were the finetuned models trained? Hyperparameters and training data for those models should be reported to be able to reproduce the finetuned baselines used in this work and to get a sense for how reasonable their training process was.**\n\nWe apologize for not including these important details about models’ finetuning. For both MARBERT and AraT5, we identified the best model on the respective development split (Dev) and blind-test on the testing split (Test). We train MARBERT for 25 epochs with a patience of 5 and AraT5 for 10 epochs. For both models, we set the learning rate of 5e-5 across all the tasks. We will add this information in Section 5 to facilitate reproducibility.\n\n**QG. Why was AraT5 omitted from the human evaluation? This seems a significant omission since it outperforms the other models by the automatic baselines.**\n\nThis is a great question! In our automated evaluation, we find that the finetuned AraT5 significantly outperforms ChatGPT (Table 2). Hence, we pick a number of tasks that are both (1) diverse and (2) cause ChatGPT to perform poorly for human evaluation (lines 528-532). Since our focus is to understand the capability of large language models on language generation in zero-shot settings, we exclude AraT5 from the human evaluation. More concretely, we focus on the LLMs that perform poorly in order to investigate whether the reason originally stems from their lack of generalizability on unseen data or the unavailability of proper automated metrics to measure their quality.\n\n**Typos Grammar Style And Presentation Improvements:**\n\n**T1. The text in the figures is too small to read without significant zooming, even on a large monitor. These font sizes should be increased for readability.**\n\nWe apologize for the inconvenience. We will improve the font size in the figures.\n\n**T2. Figures 3, 5 and 6 are not color-blind friendly color-schemes. This will make it for many readers to distinguish the categories. You can either use other color-schemes (and test them with browser extensions which simulate colorblindness) or add additional ways to delineate categories beyond the color itself.**\n\nThank you for the suggestion. We will update the mentioned figures with more color-blind-friendly schemes.\n\n**T3. Line 239 says \"deterministic and reproducible results\", but this is contradicted by the acknowledgement in the limitations that the API changes frequently. Furthermore, even at temperature 0.0, both ChatGPT and GPT-4 have been shown to not be deterministic. This line should be removed entirely. Ideally, it should be replaced by a more up-front acknowledgement about the reproducibility challenges of evaluating API-only access models rather than leaving this for the limitations.**\n\nWe thank the reviewer for the suggestion. We mentioned the API version (line 238) that we used in our work. However, we will add an acknowledgment outlining the challenge of reproducibility of the closed-models.\n\n**References**\n\n[1] Laskar et al. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets. ACL 2023.\n\n[2] Lai et al. Chatgpt beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv 2023.\n\n[3] Chung et al. Scaling Instruction-Finetuned Language Models. arXiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work evaluates large-scale commercial LLMs on existing NLP datasets to a new, widely spoken language. This gives future researchers baseline results that they can compare their methods to for a broad range of Arabic NLP tasks, while saving them the cost of querying these relatively expensive APIs.\n\n- The work evaluates across both open-source models and leading closed-source APIs for Arabic NLP. This allows more practically oriented researchers to have a sense of what the tradeoffs might be when looking at different tools for Arabic NLP.", "reasons_to_reject": "- The authors utilize a static prompt template for all tasks - but LLM prompting can result in extremely high variance and usually needs to be tuned between tasks. Without evaluating multiple prompts per task, these results may be misleading as readers have no sense what the variance is. This only compounds with the reproducibility challenges inherent to evaluations based on changing API endpoints acknowledged in the limitations. Especially since the authors draw conclusions primarily by the relative rankings of models, this work seems to have reasonable risk of some conclusions being misleading/incorrect given different prompt phrasing or as the API changes.\n\n- The work's analysis focuses on high level comparison of the performance metrics and postulations at their causes. This means that the primary takeaway from the work is that these models have significant room to grow for Arabic NLP, but they don't offer a deeper argument for how these findings should change existing research directions or what is unique about the challenges for Arabic NLP for LLMs vs. other languages.", "questions_to_authors": "A) In Section 4, you mention that an array of prompt templates was evaluated, but English prompts were found to be best. How were candidate prompts evaluated? With what dataset? On all models?\n\nB) Do you constrain the outputs of ChatGPT or BloomZ using logit bias or using logprobs rather than free-form generation? If not, how are responses matched to the label set from a free-form generation?\n\nC) You mention that your annotators are native Arabic speakers, but don't mention their familiarity with dialects. Which dialects are they familiar with and what is their level of familiarity for each dialect? Are they familiar with the non-Arabic languages in the codeswitched evaluations and if so what is their level of familiarity with them?\n\nD) How was inter-annotator reliability measured for each task evaluated with humans?\n\nE) How were the finetuned models trained? Hyperparameters and training data for those models should be reported to be able to reproduce the finetuned baselines used in this work and to get a sense for how reasonable their training process was.\n\nG) Why was AraT5 omitted from the human evaluation? This seems a significant omission since it outperforms the other models by the automatic baselines.", "paper_topic_and_main_contributions": "This work evaluates ChatGPT, BloomZ, and GPT-4 on a broad suite of NLU and NLG tasks in Arabic, covering both Dialectal and Modern Standard Arabic. It shows that these LLMs lag behind their finetuned counterparts consistently across both NLG and NLU tasks, but that GPT-4 is still a clear leader in these tasks, while open-source LLM BloomZ consistently struggles. They use both automated benchmarks and a small set of human evaluations for a subset of NLG tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The text in the figures is too small to read without significant zooming, even on a large monitor. These font sizes should be increased for readability.\n\nFigures 3, 5 and 6 are not color-blind friendly color-schemes. This will make it for many readers to distinguish the categories. You can either use other color-schemes (and test them with browser extensions which simulate colorblindness) or add additional ways to delineate categories beyond the color itself.\n\nLine 239 says \"deterministic and reproducible results\", but this is contradicted by the acknowledgement in the limitations that the API changes frequently. Furthermore, even at temperature 0.0, both ChatGPT and GPT-4 have been shown to not be deterministic. This line should be removed entirely. Ideally, it should be replaced by a more up-front acknowledgement about the reproducibility challenges of evaluating API-only access models rather than leaving this for the limitations."}}]

Author rebuttal:

Meta-review: This paper presents a large scale evaluation of the performance of ChatGPT on Modern standard Arabic and other Arabic dialects. The evaluation cover 44 distint taks and 60 datasets. 

The reviewers found this to be a valuable contribution in terms of benchmarking ChatGPT performance on Arabic tasks. There are a few concerns about the paper is too condensed giving the number of evaluation tasks and datasets. It would be great if the authors add more insights on the evaluation in the paper or appendix.
Decision: Accept-Main