JASMINE: Arabic GPT Models for Few-Shot Learning
Authors: El Moatez Billah Nagoudi
Muhammad Abdul-Mageed
AbdelRahim A. Elmadany
Alcides Alcoba Inciarte
Md Tawkat Islam Khondaker
Venue: EMNLP 2023 Main (2023)
Keywords: Arabic
Arabic dialects varieties
GPT
Few-shot learning

Abstract: Scholarship on generative pretraining (GPT) remains acutely Anglocentric, leaving serious gaps in our understanding of the whole class of autoregressive models. For example, we have little knowledge about the potential of these models and their societal impacts in diverse linguistic and cultural settings. We alleviate this issue for Arabic, a wide collection of languages and dialectal varieties with more than 400 million population, by introducing JASMINE. JASMINE is a suite of powerful Arabic autoregressive Transformer language models ranging in size between 300 million-6.7 billion parameters pretrained on a large and diverse dataset (~ 235 GB of text). We also carefully design and release a comprehensive benchmark for both automated and human evaluation of Arabic autoregressive models, with coverage of potential social biases, harms, and toxicity. Using our novel benchmark, we evaluate JASMINE extensively showing powerful performance intrinsically as well as in few-shot learning on a wide range of NLP tasks.  We aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.

JASMINE: Arabic GPT Models for Few-Shot Learning
Moatez Billah Nagoudi*. Muhammad Abdul-Maged AbdelRahim Elmadany Alcides Alcoba Inciarte* Md Tawkat Islam Khondaker
À Deep Learning& & Natural Language Processing Group, The University of British Columbia Department of Natural Language Processing & Department of Machine Learning, MBZUAI (moatez.) nagoudi, muhammad. mageed, a. e elmadany @ubc..c... ca
Abstract
Transformer language models of significant size remain largely anglocentric. This makes it difficult to bring more diverse voices to the table. Nor is it clear multilingual models such as BLOOM (Scao et al., 2022), where model capacity is split across a large number of languages and language-specific data are neither sufficiently large nor diverse, can allow equitable understanding of these models in languages other than English. It also not pos- sible study the capabilities of these models in particular linguistic eavironments (e.g.. languages of rich morphology, of diglossic nature, and/or with a large number of dialects such as Arabic) and di- verse cultural backgrounds (e.g., African, Asian, Latin American). This situation also deprives non- English communities of the rich array of benefits language model technology bring its full po- tential and emerging capabilities (Wei al., 2022) are unlocked. Alarmingly, we currently cannot study the social harms, risks, and biases associated with such models. In order to carefully investigate the risks of these models and work on preventing or at least mitigating them, we need to responsibly develop sufficiently large dedicated models outside English.
Scholarship generative pretraining (GPT) remains acutely Anglocentric. leaving serious gaps in our understanding of the whole class of autoregressive models. For example, have little knowledge about the potential of these models and their societal impacts in di- verse linguistic and cultural settings. We alle- viate this issue for Arabic, a wide collection of languages and dialectal varieties with 450 million population, by introducing JASMINE. JASMINE a suite of powerful Arabic autore- gressive Transformer language models ranging in size between 300 million-6.7 6.7 billion param- eters pretrained on large and diverse dataset (~~ 235GB of text). We also carefully design and release a comprehensive benchmark for both automated and human evaluation of Ara- bic autoregressive models, with coverage of potential social biases, harms, and toxicity. Us- ing novel benchmark, MINE extensively showing powerful perfor- mance intrinsically as well as in few-shot learn- ing wide range of NLP tasks, We aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.
evaluate JAS-
1 Introduction
To circumvent these limitations and advancc scholarship of autoregressive models beyond En- glish, we propose suite of decoder-only Trans- former models for the Arabic collection of lan- guages and language varietics. Our suite of models, dubbed JASMINE, come four different architec- tures that range in size from 300 million to 6.7 bil- lion parameters. Motivated by recent findings as to the impact of pretraining data size vis-à-vis model size (Hoffmann et al., 2022; Penedo al., 2023), carefully curate large dataset (~ 235GB of text) of high-quality text to pretrain JASMINE. Our dataset is also diverse (e.g.. covers both standard and dialectal Arabic), endowing our models with an ability to serve wider communities.
Recent work in generative pretraining (Radford et al., 2019; Brown et al., 2020; Lieber et al., 2021; Chowdhery et al., 2022; Zhang et al., 2022; Smith et al., 2022; Scao et al., 2022; Thoppilan et al., 2022: Hoffmann et al., 2022) has shown that au- toregressive models perform well on language tasks using in-context learning, without finetuning or gra- dient updates. This in-context learning approach allows models to perform new tasks with only simple instructions and few optional examples, which can be further improved by model adapta- tion through prompt tuning (Lester et al., 2021), In spite of this progress, autoregressive pretrained
Authors contributed equally.
Our work also fills another significant gap for

---

Arabic autoregressive models, i... that of an evalua- tion benchmark. We introduce an evaluation bench- mark comprising a wide collection of test datasets and protocols. Using our benchmark, we evalu- ate JASMINE extensively both intrinsically (using perplexity) and extrinsically (c.g., on few-shot set- tings). Our evaluation demonstrates the superiority of JASMINE compared to available baselines. We also perform human evaluations to investigate the ability of our models write fluent and coherent standard as well as dialectal Arabic across various domains (c.g., news, literary, Twitter). Our eval- uations reveal that our JASMINE models posses powerful representations, allowing them to excel in few-shot learning and produce outputs that can be identified by humans only at chance level. Since autoregressive models often carry social biases, harms, and toxicity, our evaluation testbed involves the creation of a set of carefully-designa datasets for measuring a range of social risks. Addition- ally, we aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.
is about related work. We conclude in Section 8.
JASMINE
2.1 Arabic
Arabic is collection of languages and language varicties, some of which (c.g., Moroccan Arabic and Egyptian Arabic) are not mutually intelligible. Classical Arabic (CA) is the variety used in old Ara- bic poetry and the Qur' an, and is employed side by side with other varieties to date. Modern Standard Arabic (MSA) is a more modern variety (Badawi, 1973) of Arabic that is usually used in pan-Arab media, government, and formal education across the Arab world. Dialectal Arabic (DA) is the term used to refer to Arabic dialects. Dialects are some- times defined regionally (c.g., Gulf, Levantine, Nile Basin, and North African (Habash, 2010; Abdul- Mageed, 2015)), but also at the country or even province levels (e.g., (Bouamor et al., 2018; Abdul- Mageed et al., 2020b,a, 2021b, 2022)). We now introduce JASMINE.
2.2 (Pretraining) Data
To summarize, we offer the following contribu- tions: (1) We develop JASMINE, a suite of four autoregressive language models for Arabic, rang- ing in size between 300 million to 6.7 billion pa- rameters pretrained with diverse dataset. (2) We evaluate JASMINE extensively, introducing a com- prehensive evaluation benchmark for wide range of NLP tasks. We demonstrate JASMINE's ability to write fluent language and learn well in-context across rich contexts in few-shot settings. (3) Our evaluation benchmark involves the creation and release of datasets for investigating potential so- cial biases, harms, and toxicity. Based these evaluations, we join arms in calling for ethical practices when working with language models and inviting future research on mitigating their social risks. (4) We aim to responsibly and gradually re- lease our models with interested researchers, along with code for experimenting with them, hoping our work will trigger applications and further research in understanding autoregressive models outside En- glish.
Our dataset is linguistically diverse, covering all categories of Arabic (i.e., CA, DA, and MSA), as we will now describe.
CA Data. We use the Open Islamicate Texts Initia- tive (OpenITI) corpus (v1.6) (Nigst et al., 2020).' OpenI'I contains 11, 195 premodern Islamic books mainly collected from Shamela Liberay, Al-Jami Al-Kabir collection (JK), books digitized by Jorda- nian publisher Markaz AI- Al-Turäth, and the Shia Li- brary. MSA Data. We use 223 GB of MSA text (23.7 billion tokens) from the following sources: AraNews,2 (Nagoudi et al., 2020), EI-Khair (EI- Khair, 2016), Gigaword, OSCAR (Suárez et al., 2019), OSIAN (Zeroual et al., 2019), Wikipedia Arabic, and Hindawi Books. 6 We also extract the Arabic part of the multilingual Colossal Clean Crawled Corpus (mC4) (Xue et al., 2020) and clean it (see $ 2.3 for cleaning procedure). We call the extracted portion AraC4 (more details are in Ap- pendix A.2). Dialectal Data (DA). We use cor- pus of 1.5 billion Arabic tweets (178GB) randomly
The rest of the paper is organized as follows: We introduce JASMINE in Section 2, describe our evaluation strategies in Section 3, and our evalua- tion benchmark in Section 4. In Section 5, we offer human evaluations model output. Section 6 is an analysis of social bias in the model, and Section 7
We exclude random sample of 1K books from OpenITI for later use evaluating JASMINE perplexity (see $ 4.1. hhttps://hhmmlla.. ws.
http::/kkitab http:://kiaabppojjcct.o wtwppeee...g//oo/ooolllll
"huups//stiaooniieiibbaaycce
/httss://catalog Ide upenn LDC200990.
"htts::/wwww.i hindawi.org//onne
2

---

Source
Size
Tokens
Model
Layers
Heads
Embed
Seq
Parameterss
EAS
12
12
768
2,018
350M
AraC4
173GB
19.8B
JASMINE1.1
2,045
2. 040
1.3B
AraNews
18.3GB
1.8B
JASMINE2,70
2,500
2. 048
2.7B
El-Khair
16GB
1.6B
JASMINE4.75
32
32
4,006
2,048
6.7B
Hindawi,2
1.1GB
78.6M
Table 2: Parameter values for our JASMINE: models.
Gigawords
1.1B
OSIAN
2.8GB
292.6M
OSCAR-Ege
32MB
3.8M
2.4 Model Design and Implementation
Wiki
1.6GB
156.5M
We exploit our diverse dataset to train four different variants of JASMINE, follows: JASMINESSIM: JASMINE JASMINE2.1H and JA JASMINE,.7B. We pretrain JASMINE mod- els for 500k steps each using the autoregressive next-step prediction objective (Radford et al., 2019) and the Transformer-basee GPT-Neo (Black et al., 2021) replication of the GPT-3 (Brown et al., 2020) architecture. Details of the various architectures of JASMINE are in Table 2.
MSA-Total
222.8GB
23.7B
CA
12GB
1.1B
MSA+CA
243.8GB
24.8B
Twitter
178GB
21.9B
Table 1: Datasets used in JASMINE models.
sampled from a large in- ho-souee dataset of 13 bil- lion Arabic tweets. This dataset is used only for finetuning one of our models (see Scction 5), rather than pretraining.
3 Evaluation Strategies
We follow previous literature (Brown al., 2020; Howcroft et al., 2020; Zhang al., 2022) in evalu- ating our models extensively, under both intrinsic and extrinsic conditions as we now explain.
Data Distribution. We analyze the distribution of MSA DA in both our AraC4 and Twitter collections using SoTA binary classifier (Abdul- Mageed al., 2021a) (MSA vs. dialect, ~ 88% Fi) on a random sample of 100 million samples from each. We find that our Twitter data involves 28.39% predicted dialect tweets and our AraC4 data in- volves 5.7% predicted dialect sentences. We then run another SoTA country-level classifier (Abdul- Mageed al., 2021 la) (~ 40% Fi) on the predicted dialect portions from each dataset, finding that Twitter data more diverse than AraC4. For exam- ple, our classifier tags 80% of the predicted AraC4 dialects as Egyptian, 2.86% as Bahraini, 1.85% as Libyan, leaving other dialects to be only marginally represented. Refer to Table for more information about our pretraining data (e.g., size, number of tokens) and Table A.1 for country-level predicted dialects from each of the datasets.
Intrinsic Evaluation. Perplexity (PPL) is a widely used metric that estimates how well a language model predicts given text. For a tokenized text T (w1, W1, ..... Wn), perplexity of T is:
PPL(T) = expf Σ log po(w:lwei) (1)
Where log (w; ww2)) is the logelikelioood of the ith word conditioned on the previous words Wei- Extrinsic Evaluation. We employ three settings: (1) few-shot, where a model is given & examples de- scribing the task at inference time as conditioning, but without updating the models' weights. (2) one- shot, which is the same as few-shot except that only one example is provided to the model (i.e., k=1). (3) zero-shot, where no demonstrations provided to the model (i.e., k=0).
2.3 Preprocessing and Vocabulary
We clean our pretraining data by removing HTML tags, elongation, and hash signs. We also reduce repetitive characters, emojis, and emoticons to only two occurrences per instance. Further, we replace URLs and user mentions with the <URL> and <USER> strings. To create our vocabulary, we use a BPE--based tokenizer similar GPT-2 (Radford et al., 2019), with a vocabulary of 64, 000 BPE tokens. Refer to Appendix A. for more details.
Evaluation Benchmark
We evaluate JASMINE on 23 different datasets, representing five different tasks: language model- ing, autocompletion, commonsense inference, word manipulation, and natural language understanding. We now introduce each of these tasks along with related datasets.
"The number of parameters is suffixed to model names.
3

---

4.1 Language Modeling
Medel
Ara
Williing
AraNews
WTQ4
Wili
Op-171
Aug.
ArsGPT22p
87.65
65.27
34.22
41.23
595.71
151.81
119.50
As explained, we calculate the perplexity of our models as intrinsic evaluation. Since there is no standard dataset for evaluating perplexity on Ara- bic texts, create and release new multi-domain dataset totaling 6K documents extracted from six publicly available sources. These datasets are not our pretraining and cover three Arabic vari- eties: MSA, dialect, and CA. We introduce each of them. (1) Arabic Wikipedia. We select 1K articles from Arabic Wikipedia (AraWiki), pub- lished after October 2022 to avoid leakage with our (2) WikiLingua. Introduced by Faisal Lad- hak and McKeown (2020), this resource contains article and summary pairs in 18 languages, includ- ing Arabic, extracted from WikiHow. We extract 1K Arabic articles from the test set of WikiLin- gua. (3) News Articles. We collect 1K news articles from ~ 100 Arabic online sources. The ar- ticles are not in our pretraining and cover different domains (e.g., culture, economy, politics, sports). (4) Watan2004. We select 1K articles from an old dataset, Watan2004 (WT04) (Abbas et al., 2011), For dialectal and classical Arabic, we also extract a random 1K articles from each of the following sources: (5) Egy Egyptian Arabic articles from Wikipedia dumps, and (6) CA-Book. Open Islamicate Texts Initiative (OpenITI) corpus (Nigst et al., 2020).
67.57
ArsGPT22us
51.57
49.45
30.6
32.15
595.07
122
103.08
mGPT
3041.66
122075
1936
156501
140.78
146.67
16137
JASMINFSARM
JASMINE
35.73
35.06
15.45
27.05
105.53
51.14
15.78
24.73
91.71
51.95
JASMINE,M
50.27
31.21
16.12
23. 85
67.
77.52
42.25
Table 3: Results the perplexity our JASMINE mod- els on our language modeling benchmark. We compare to AraGPT2 (Antoun et al., 2020) and mGPT (Shliazhko al., 2022).
Medele
0-shot
S-chots
16-shots
24-shots
ArnGPT2re
11.13
10.38
12:17
12.19
12.82
ArGGPT2/0s
10.86
12.78
13.77
13.18
ArrGPT2ne
13.61
15.21
16.74
19:33
11:11
Neus Tille
Ar
14.92
11.51
17.00
10.89
mGPT,
12.80
13.63
10:32
10:48
10.34
JASMINEsww
12.79
13.39
16.09
18.04
16.67
JASMINE
15.25
16.13
17.49
20.98
16.01
JASMINEs
15.88
16.93
17.57
23.13
15.82
JASMINE
15.91
17.44
18:41
24.10
17.96
Table 4: Zero-, one-, and few-shot performance in F1 on the news title completion tasks.
out the last word and ask it to predict the masked word. We experiment with our models under zero-, one- and few-shot settings. Results. Table shows results on the news title datasets, and we provide results for the two other autocompletion datasets in Table C.I. From Table 4 we can see that JAS- MINE models perform best in all settings." 10 We also observe that more demonstrations tend to help improve performance. We also note that the mod- els achieve the best autocompletion on the news stories subtask, perhaps due to our pretraining data involving significant amounts of news. The models also perform reasonably well on the theses titles domain, perhaps since our pretraining datasets in- volve specialized books covering academic topics. We notice drop in model performance under the 24-shot setting, perhaps since few-shot learning can be sensitive to the order of the shots Wei et al. (2021); Brown et al. (2020); Lu et al. (2022).
Results. Table 3 shows the zero-shot BPE- token level perplexity of our JASMINE models on the six datasets. We compare to the four AraGPT2 models proposed by Antoun et al. (2021) and mGPT (Shliazhko et al., 2022) as baselines. Our JASMINE models clearly outperform all base- lines by significant margin, with JASMINE6.7 reaching an average PPL of 42.25.
4.2 Autocompletion
The goal of autocompletion is to predict the last word for a given text. For this, we create a dataset totaling 15K samples. These are news headlines (5K phrases/sentences), news stories (5K para- graphs), and theses titles (5K phrases/sentences)) All samples are collected from diverse online sources. For example, the thesis titles cover do- mains such us الAدارة (management), القسس je (psy- chology), and القاورن (law). For evaluation, give JASMINE a prompt (title paragraph) with-
4.3 Commonsense Inference
Since there is no Arabic commonsense inference evaluation dataset, we follow methods introduced by Zellers al. (2018) to create new, high-qualiy Arabic commonsense collection using a random
1OFor this and upcoming experiments, we restrict evaluation to our smaller models (all or any of our 1.3B-6.78 models) due to constraints on our computing resources.
httpp://wwwwwwkkhow..oo "htnpss/huggigggfacc.. htt.:/gggn...g.oooooooddeeeeeeae lingua.
4

---

generated end- sample of 16, 707 examples from Arabic WikiHow, Each example has a context and a correct answer." For each context, we create three generated answers using an adversarial approach. We refer to our new dataset as AraSWAG (Arabic Situations With Adversarial Generations). We next provide a full explanation of it.
Caroa
Ex:igg
Gurld Facket
Inallgg
Gorld
lakee
Then the clasaieer
doc dscomssn sate
Caaca
MARBERT
Rel
cerd
Colaod
Rail Bribg
Goril
Goeld
Golld
Eadiy
Initial Dataset Creation. 10K examples from Arabic WikiHow. 12 We then finetune AraT5 (Nagoudi et al., 2022) on the sam- pled examples separately, where we feed the model with the contexts in order to generate the endings, After finetuning, we generate three possible end- ings for different set of WikiHow (17K examples), We generate the ending by setting tops 50 and topp = 0.95 to mimic human-like writings. There- fore, our initial datasets contain one context and four endings (one real and three generated).
We randomly sample
D
Cantes
Indeg]
tninn?
AraTS
Replace masil peetean with adeersanaa
Figure 1: Overview of AraSWAG dataset creation. On each iteration, new MARBERT trained on dummy training set Diruis. to identify casily-classifeed gener- ated endings on the dummy test set Drest- The finetuned AraT5 is used to replace ings with adversarial ones. This process is repeated iteratively to obtain challenging dataset.
Adversarial Dataset Creation. To make the com- monsense inference task more challenging, we fol- low (Zellers et al., 2018, 2019) and apply the adver- sarial filtering (AF) method on the initial dataset. Specifically, on each iteration, the dataset is ran- domly partitioned into Dirain and Diest with split of 8:2. We then finetune a MARBERT (Abdul- Mageed al., 2021a) model in order to classify endings as real generated on Derain. We evalu- ate the finetuned model on Diest, then apply AF to replace casy-to-classify generations in Drost with newly generated endings using the finetuned AraT5, This process continues until accuracy of these ad- versaries converges. We observe that during con- vergence, the accuracy of MARBERT drops to 30%. Finally, randomly split the resulting AraSWAG dataset into training (Train=14, 288), validation (Dev= 44), and testing (Test=1, 675) sets.
AraSwag Prompting Example
Context:
de احيي الفخصية نطاايي عن اررمي راسسيي tiee Wine حاول الرمم من بK lسطط اللهم 3f الريد الونن السيية جياا الو عن
sus بمة للبفاء البااية المارعن مارسي
Ending 1: gou غغاي نخام عن راطفاط بالفطة حم
Ending
السروا و الله من أوواا ادري الأفير وي وزاه القاصس فال عككل ما امررق
Ending 3
Ending 4 تعلل تيه علوو طييل الر كدفي
Answer:
إلسواال الل من أويي الرري الأهر رف
Table 5: A context and four endings from AraSWAG, with the second ending as a correct answer.
AraSWAG. Results. As Table 6 shows, although our dataset is challenging, JASMINE2.1 signifi- cantly outperforms baselines (37.18 Fj).
We use AraSWAG to seed our 350B, 1.3B, and 2.7B JASMINE models and the baselines with context and four endings, one original (true) and three generated (false) as explained. We then com- pute for each ending a language modeling score (LMS), following Nadeem et al. (2021), 13 to iden- tify whether it is related to the seed context or not. We evaluate the likelihood of each candidate's 's ending conditioned on the context and choose the candidate with the highest LMS. Table 5 shows example of a context and four endings from
4.4 Word Manipulation
We test our JASMINE models' ability to learn how to correct word-level errors (i.e., recover the orig- inal word) from a few examples. For this, we ex- ploit one existing and one new dataset: (i) Natural Spelling Errors. We use QALB (Zaghouani et al., 201 4), large marrally collection of Ara- bic sentences. QALB covers variety of types of errors, from which extract 22.8k words with spelling errors and errors proper names. (ii) Synthetic Errors. We create synthetic dataset with five scrambling tasks using the same method introduced in GPT-3 (Radford et al., 2019). The
II "htps://www..a hntps//wwwwwwikthow.co 12 hhttps:/wwwwwikkiii wwtp://wwwwwwkoi..coo
13 Refer to Appendix B.1 for details about LMS.
5

---

Models
Acc
Fi
Manipulation
Original
Manipulated
AraGPT2ps
23.64
23.61
CL
AraGPT2m0
28.23
28.23
A1
اااتامم
AraGPT2>9M
32.59
32.03
AraGPT21.8
26.74
26.75
A2
الرااية
L Alll
JASMINEFsmm
28.23
28.23
RI
الyرل
w st
JASMINE,,.
35.28
35.26
JASMINE2.5
37.23
37.18
RW
Jlاh
Table 6: Performance on the AraSWAG dataset.
Table 8: A sample of word errors generated using ma- chine manipulated approach. CL: Cycle Letters. A1:
Sottice
AraGiF12.
JARMINNiies
JASMINiLus
JASMINEL
JASMINEGs
Anagrams 1. A2: Anagrams 2. RI: Random Insertion. RW: Reversed Words.
01111
0.13
0.1
2023) under zero-, one-, and few-shots conditions. Performance of JASMINE on ORCA is shown in Table C.2. We find that JASMINE,,71 acquires the best results, again clearly outperforming all baselines.
0.2
060
163
121
2/8
138
301
057
3.5
137
732
K34
5-b.1
240
0.59
1853
912
16:30
150
100
325
139
544
1857
130
144
104
932
11.6
Human Evaluation of Model Output
6/6
035
6:5
6.18
038
035
Idba
112
057
634
124
128
144
We carry out set of human studies investigate the ability of our JASMINE model generate texts from diverse domains. This includes the news, literary (ie.. poetry), and Twitter domains. We also investigate the ability of the same model to produce dialectal continuations when seeded by sequences from the respective dialects. We provide sample generations from these experiments in Table 9. News Story Generation. We sample 10 news arti- cles from each of 10 categories of a news dataset not in our pretraining (total=100 articles). 14 For each news category, we extract the first sentence from five sampled articles and use the sentence to prompt our model to generate an output for each article. We then provide the 50 JASMINE2,11 generated texts and the remaining 50 original arti- cles two college-educatd
CL
5:bax
511
159
380
437
334
600
16:90
762
154
359
435
346
624
26.6
135
133
433
334
6.4
Table 7: Performance on the different word scrambling tasks (F)). We exclude results for reversed words from the table since, similar to GPT-3, the models did not predict any correct answers (i.e., Fj=0).
tasks are (1) cycle letters (CL), where the model is given a word with its letters cycled. (2) ana- grams/ (A7), where every letter in the word except the first and last are scrambled randomly. (3) ana- grams2 (A2), where every letter in the word except the two first and last letters are scrambled randomly. (4) random insertion (RI), where a random space character or punctuation is inserted between each letter of word. (5) reversed words (RW), where we task the model to recover the backward version of the word. Table 8 offers an illustrative example for each word scrambling technique. For each of the five techniques, we generate 10K top words from a dictionary extracted from Wikipedia Arabic and Hindawi Books. Results. As Table 7 shows, our models achieve better results in 23 out of 25 settings.
Arabic native speak- ers to assign a label from the set (human, gener- ated) at the article level. We find that annotators only have a random chance to identify generations by our model. In fact, for the 50 articles generated by our model, either of the two annotators could identify only 11 samples (i.c., 22%) and the two an- notators never agreed on any of the samples. This shows that our model is able output sensible, human-like language for the news domain. We pro-
4.5 Evaluation Arabic NLU Benchmark
14The categories are from set [Economy, Education, Health, History, Media, Politics, Religion. Sports, Technology, Weather), and the average size an article is 125 words. WW shuftle the generated and the original articles.
We also investigate the capability of our models on six text classification datasets from the large and diverse ORCA benchmark (Elmadany et al.,
6

---

Emply prompt
on an in-house dataset of 1.5 billion tweets for 100k steps, restricting the sequence length to 128 BPE tokens and adding the prefix "sjá"" ("write a tweet:") to all tweets. We refer to the resulting model as JASMINE Aweet and provide samples from its oulput in Table E.4. A gold annotation study reveals that humans are able to detect generations from JASMINE only in 48.53% of the time, thus reflecting the model's s ability output high- quality tweets.
شروا uejb eir is iiti الأمم مoر onn atdy je ds lyor
A yلsيي wr dاeee is Ail zu st tut اes Lmbog مes ept fifh مالون اللهي بنب en الكيي in الهيي ماك yo sht في is الت رنك sn cake similly wulf set dh الساا أن ف مي a- بااlo س juSS Join الصال نساع ت ايس .. موتتت الننس aatt la 18/5
Part partry's tine prompi
--r بيل gيt oh good di I بaي Y t ألوو wiب الننس م su diple ha ley aithi to اiي of % f o with قي of Ho an aafe was
of peetry prompt
Dialectal Generation. We study our model's s abil- ity to generate dialectal texts by seeding it se- quences from a new Arabic dialects dataset that we prepare. We create the dataset by manually transcribing a total of 500 speech utterances from five different Arabic dialects from the set /Algeria, Egypt, Jordan, Morocco, Yemen/ (100 utterances, around 30 seconds long from each dialect). !7 We acquire 500 outputs from our model by seeding the transcriptions sample under one-shot, refer- ring to the dataset as STGen. Appendix Table E.7 shows samples from these dialect-prompted gener- ations.
dsm off Cillige of fite اله 3575 you Jlell o d43 Je to of shlll Uoe byet
مos tl de
wot A you cel aut الل ( the maphe lae of bo whlll you is
so potho I shall with evle anw uie gilll jile on by so paisn المt of
Diatect
Pugh to ماي it تعلس عداا غحس vn معام الون م بي+ الل البوال حازة ب the ley per مويوي و حرية مساe اكر طايي اt s Sy inl yf dise ال لن بم منز ددد وييي مم is معs if jez dele bi حيي بنن منن wimm اiهl زنر
dنs مزو wh 2o
Tweets
-الlle الاات دلر دايي + gh t معd wللب ss Jos of let wieil العع اesst العمl pas tyely pas oft ooe wing Jull so Aplle oa الررا الميي مisم y fusn pa aye jusel pou
Annotation and Results. We ask annotators with native fluency in the five dialects mentioned to sign labels in two stages: MSA vs. dialect (stage one); and if dialect, whether the dialect the same as the seed utterance (stage two). We find that an- notators assign a dialect tag 52.86% of the time, with the model staying within the same dialect as the prompt utterance 45.37% of the time. We also find that while the model excels on sticking to the Egyptian dialect of prompt (79.35%), it is less successful in doing so for Jordanian, Moroc- can, Yemeni, and Algerian (with 47. 17.62%, 48.39%, 4.35%, 47. 17., respectively). We hypothesize that this is a function of the model seeing larger amounts of Egyptian dialects and the overlap be- tween MSA and dialects." 18 We also make an excit- ing discovery in the context of this experiment: the model generates multi- multi-party dialect conversations (see Table F.7).
vet oite cliss ta
Table 9: Examples of generated 'poems', Egyptian di- alect, and tweets from JASMINE 2.78. We color the initial prompt with gray.
vide sample generations from this experiment in Table E.2.
Poetry Generation. We experiment with seeding our model with three lines of real poetry at a time (3-shot) and find that while generated sequences do look like 'poetry', the model is not able to con- sistently maintain the rhyme. We show the results of this experiment in Table E.5. We then run an- other experiment where we collect a poetry dataset of ~ 22K poems 16 and further pretrain the model with it for ~ 50k steps. We refer to the resulting model as JASMINEpeetry and provide samples from its output in Table E.6. A human annota- tion study reveals that annotators are able to tease apart JASMINEpenn generations from human po- ctry in 52.63% of the time. We note, however, that model generations are quite sensible and it is able to keep the rhyme in many output lines.
Analysis of Social Bias
While autoregressive models are able to produce fluent texts which have multitude of useful ap- plications, they can also carry societal biases. To
We provide full details of our new speech transcription dataset in Appendix B.3.
Tweet Generation. We experiment with teaching model to write tweets by further pretraining 16 Details of the dataset are in Appendix B.2.
"We hypothesize that if we seed the model with longer sequences it will he abler to stay within the same dialect as the sced, and cast this as future research.
7

---

quantify biases in our generative models, we use conditional generation (i.e., autocomplete genera- tion) (Shwartz et al., 2020; Brown et al., 2020). For all social bias experiments, we use JASMINE277. We provide sample outputs from all these experi- ments in Table E.3.
Biggnees Boferse BHASS
Biases in Gender Autocompletion. We inves- tigate associations between occupation and lin- guistic gender by prompting the model. For this cause, manually prepare list of 100 occupa- tions which we use with the following template: "The <occupation> is often practiced by..." ..." (e-g.. lerle la غال Cللlب We provide the full list Ta- ble E.
Figure 2: Percentages of correlates bias towards reli- gions/ideologies and religious/ideologiaal groups.
Results. We find that 62.50% of the 100 occu- pations we test are more likely to be followed by male linguistic gender. This means that the model is male-lcaning when occupation context is given.
wage occupations 100% of the time. When the context is Africa, region information triggers very biased association: people of African descent are associated with low-wage occupations 100% of the time. Again, these findings confirm what we know-autoregrran autoregressive models, even those trained on diverse data (e.g., not only from the web but also from books), suffer from various types of biases. Religion and Religious Groups. To eval- uate potential biases towards given reli- gion/ideology or religious/ideologicall group, we introduce the following template to construct our prompts "These <R> guys always are... (Lta LĀs الدة الغباب Nحه),, where either one of the four religions/ideologiee. Atheists, Islam, Ju- daism, Christianity, and Sikh and one of seven Mus- lim/Islamic groups from the set /Ash aris, Salafis, Muslim Brotherhood, Shi' a, Sufis, Sunni). Again, top-k and top-p sampling (with k=50 and p=0.95) to generate 50 completions for each of the 12 prompts. Then, we measure whether or not the generated texts are abusive, dangerous, hateful, or offensive using four SoTA classifiers (one for each task) from Abdul-Mageed et al. (2021a). Results. We present results in Figure 2. We observe that dangerous language is predicted as most associated with Atheists; and offensive language is most asso- ciated with Atheist, Shiite, and Jewish groups. The model associates hateful language equally to Sunni and Shiite groups. Importantly, we believe this analysis bias should be considered with caution. Human Analysis. We augment automated analysis of religious and ideological bias with a human study where ask two native speakers label 400 random classifier outputs, finding the two annotators to agree with the classifiers as follows: 86.50 (dangerous). 81.00 (hateful), and 77.50 (of-
Gender, Color, and Region. Inspired by Kirk al. (2021), we use the following template "You always find [X][Y][Z] working as ... where X is binary gender, in the set [Africa, Asia, America, Europe), and Z represents one of two colors black or white. This gives us total of 16 prompt combina- tions. One example from this combination can be a عماون السود المرععيونن Jلرا )l 2ه 'o LAb (En- glish: "You't ' always find black American men working as as...") Then, we use top-k and top-p sampling (with top-k=50 and top-p=0.95) to gen- crate 10 completions for each of the 16 prompt combinations, this gives us 1, 600 generated sen- tences of which we keep only 1, 000 sentences that contain professions. Finally, we manually clas- sify the generated sequences into one of three cate- gories from the manually prepared set (high-wage, medium-wage, low-wage).
one of the regions
Results. We manually analyze our model output and find that white people are associated with high- wage jobs 51.25% of the time and medium-wage jobs 75% of the time (zero association with low-paying jobs). In contrast, 72.50% of people of color are associated with medium- waadium--waeeeee profes- with high-wage professions (with the remaining 3.75% associated with low- wage jobs). These results show that the model carries social biases related to color. We also find that these biases are worse when we consider com- binations of color, region, and gender. For example, European white people are associated with high-
sions and only 23. 75%
8

---

prompts, then the authors rank the output with hu- man feedback to align the model with the users' in- tent. Later, CharGP719 followed the same training procedure to develop a conversational agent. Taori et al. (2023) finetuned an instruction-following lan- guage model, Alpaca, with LLaMA as the backbone model 52K generated instruction instructions based fensive). We take these high agreements to mean that we can depend on the SoTA classifiers for anal- ysis of bias our particular case. We provide more details about the human annotation guidelines in Appendix E.2.
7 Related Work
on Wang et al. (2022). Anand et al. (2023) develop chatbot massive curated corpus created us- ing GPT-3.5-Turbo. Geng et al. (2023) fine-tune LLaMA, Koala on data scraped from the web. Con- currently, Chiang et al. (2023) introduce Vicuna using GPT-4 (OpenAI, 2023) to assess and rank the outputs. Besides, several other models have been released based on instruction-tumg tunnng (e.g., Dolly) and RL (e.g., OpenAssistant). 21
Large Language Models (LLMs).. Brown et al. (2020) develop GPT-3 and show its abilities few-shot learning. Several other works followed, usually introducing larger models (Rae al., 2021; Thoppilan et al., 2022: Smith et al.. 2022). By way of examples, PaL- (Chowdhery et al., 2022) is a 540B densely activated, autoregressive Trans- former model trained on 780B tokens. Chowdhery et al. (2022) demonstrate continued benefits of scal- ing by achieving SOTA few-shot learning results on hundreds of NLU and NLG tasks. Zhang al. (2022) introduce OPT and seeks enable repro- ducible and responsible research at scale. Smith et al. (2022) train Megatron-Turing NLG with 530B parameters. A number of recent works such as TO (Sanh et al.. 2021), FLAN (Wei et al., 2021), and BLOOM (Scao et al., 2022) focus on directly improving language model's s zero-shot learning ca- pabilities through large-scale multitask finetuning. More recently. Touvron et al. (2023) introduce a large efficient model called LLaMA trained on tril- lions of tokens from publicly accessible datasets. Language Model Alignment. Ziegler et al. (2019); Stiemnon et al. (2020); Wu et al. (2021) apply re- inforcement learning to align language models for text summarization. Similarly, human feedback has been used to align language models for di- alogue generation (Jaques et al., 2019; Hancock et al., 2019), story generation (Zhou and Xu, 2020), evidence extraction (Perez et al., 2019). Most re- cently, Madaan et al. (2022) use written human feedback to augment prompts and improve the per- formance of GPT-3. Glaese et al. (2022) introduce Sparrow, a model trained to be more helpful, cor- rect, and harmless compared prompted language models.
Ethics and Bias in Language Models. The recent success of LLMs is associated with various poten- tial risks since the web pretraining datasets them- selves are biased (Bender et al., 2021; Bommasani et al., 2021; De-Arteaga et al., 2019; Dodge et al., 2021). Magar and Schwartz (2022); Tal et al. (2022) show that the risk of biases gets higher with the in- crease of the model size, causing biases to resurface during the downstream tasks such NLI (Poliak et al., 2018; Sharma al., 2021), coreference res- olution (Rudinger et al., 2018; Zhao al., 2018), and MT (Stanovsky et al., 2019). A number of ethical considerations related to PLMs have been studied, including memorizing and revealing pri- vate information (Carlini et al., 2022), or spreading misinformation (Weidinger et al., 2021).
Conclusion
We introduced JASMINE, suite of powerful GPT models for Arabic varying in size between 300 million 6.7 billion parameters. Our models are pretrained on large dataset of diverse Arabic vari- ctics from multiple domains. We also introduced a novel evaluation benchmark for Arabic GPT mod- els. Using our benchmark, we demonstrate how it is that our models excel in few-shot learning as well as producing fluent texts that humans can only de- tect at chance level. We plan to responsibly release our models with researchers to support scholarship in this important research area.
Instruction-unning of LLMs. Weller et al. (2020) introduce framework, ZEST, to solve new task after reading its description. Schick and Schütze (2021) develop a novel pattern exploiting training (PET) scheme to verbalize supervised classifica- tion task into cloze question format. Recently, Ouyang et al. (2022) propose InstrucrGPT, where the authors first finetune GPT-3 with labeler-written
httpp://ggauuu..comdaaaaaa hbbb:///hh..commmmbbaaaaaabbbbly bhtpss//opeen-assistaatttt

---

9 Limitations
time and are not energy efficient. We acknowledge this important issue and believe work on creating energy-efficient models should continue to receive scholarly attention.
We identify the following limitations in our work:
1. Although we strive to include as much dialec- tal texts in our pretraining data as is possible, our automated analysis reveals that the dataset still does not have wide coverage of some dialects such as Algerian, Iraqi, Moroccan, Sudanese, Syrian, and Yemeni. One way to improve JASMINE performance on dialectal generation would be to collect more data from these varieties and further pretrain the models with this new collection.
Data. Our pretraining datasets are collected from the public domain and cover diverse genres, com- munities, and varieties of Arabic. As we have demonstrated, our JASMINE models have the po- tential to power applications involving several vari- eties of Arabic and serve wide populations.
Data Copyright. We emphasize that all the datasets (CA, DA, and MSA) we use are collected from publicly available sources. We confirm that our data collection does not violate the copyrights of any of these sources. This includes X (previously Twitter). We would also like to emphasize that all our base models (sizes 300M, 1.3B, 2.7B, and 6. 6.7B) are pretrained without use of X/Twitter data. As such, all of these four base models can be shared with others responsibly with no concerns related to Twitter data use. More preciscly, we use 1.5B tweets to further pretrain only of these base models (JASMINE 2.7B parameters) test the model's 's ability to generate sensible 'tweets'. Model Release. We plan release our mod- els only responsibly. We will set stricter con- ditions on releasing the model finetuned on tweets, JASMINE Namely, we will require that this model not be deployed in real-world and not be shared publicly.
2. Although some works in the literature use word lists to remove toxic and hateful lan- guage from the pretraining data, we do not follow this practice. The reason is that we wanted our models to be suited for use in toxic and hateful language detection as few shot learners. We also believe that use of word lists, although can be useful in removing some anti-social content, can also be only cosmetic when it comes data cleaning. Regardless, we believe our models should be utilized with caution and approaches to mitigating social risks, biases, and toxicities should be carefully applied.
3. One of the disadvantages of autoregressive models in general is that they can be mis- used for generating fake content or even be de- ployed for producing misinformation at scale. This is is one of the most dangerous uses of this class of models. For these reasons, we believe all necessary measures ought to be taken around their use and JASMINE is no exception. This may include, for example, regulations and policies that restrict these to pro-social use such as in education, travel, recreation, etc. Due to these concerns, we will release our models only responsibly. For ex- ample, we will require users requesting our models to provide information about intended uses. We will also encourage use of our mod- cls in research secking to mitigate social bi- ases in LMS, develop new mitigation methods, etc.
Privacy, JASMINE is developed using publicly available data. Hence, we do not have serious con- cerns about personal information being retrievable from our trained models. To alleviate concerns about privacy in tweets used in JASMINE note that we removed tweet IDs, all user- names, and URLs before pretraining the model. Again, JASMINE will only be released under strict conditions.
Human Annotation. The human annotators in- volved in this project are two of the authors of this paper. Both annotators are Arabic native speakers holding Ph.D. degrees with extensive experience in NLP. They are full-time employees of the re- search group responsible for this work, and data annotation is part of their job duties. No Institu- tional Review Board (IRB) review or approval was required for this project since we only use publicly available data, which does not require access to any social networking account or password. In addition, no external annotators were involved in this work.
10 Ethics Statement
Energy efficiency. Our JASMINE models, similar to many large PLMs, needed significant pretraining
10

---

Bias Analysis. The goal of our bias analysis is to determine whether any biases related to "gender", "color", or "region" exist. For instance, color has historically been a significant cause of social injus- tice and remains relevant in many societies today. We find it challenging to study bias in models with- out referencing the concept of "color". However, we would like to highlight that the term "color" is sensitive and recommend avoiding potentially dis- criminatory terms whenever possible. We clearly note our respect for sensitivities surrounding this concept.
References
Mourad Abbas, Kamel Smaili, and Daoud Berkani. 2011. Evaluation of topic identification methods arabic corpora. JDIM, 9(5):885 192.
Muhammad Abdul-Magedd 2015. Subjectivity and senfiment analysis of Arabic morephologically- rich language. Ph.D. thesis, Indiana University.
Muhammad Abdul-Mageed, AbdelRahim Elmadany, and Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Ara- bic. In Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 7088-7105, Online. Association for Computational Linguistics.
Applications. Similar to many autoregressive lan- guage models, JASMINE can be misused. Mean- while, JASMINE can be deployed for wide host of useful applications such as in education and health.
Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a. NADI 2020: The first nuanced Arabic dialect identification shared task. Proceedings of the Fifth Arabie Natu- ral Language Processing Workshop, pages 97-1 Barcelona, Spain (Online). Association for Computa- tional Linguisties.
A Acknowledgements
We gratefully acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Hu- manities Research Council of Canada (SSHRC;
Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabie Natural Language Processing Workshop, pages 244-259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
435-2018-0576: 895-2020-1004; 895-2021-0008), Canadian Foundation for Innovation 37771),, Digital Research Alliance of Canada, 22 and UBC ARC-Sockeye." 23 We thank the Google TFRC pro- gram for providing us with free TPU access.
Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. Nadi 2022: The third nuanced arabic dialect identifi- cation shared task. arXiv preprint XXix:2220009::
Muhammad Abdul- Magule, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020b. Toward micro- dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 5855-5876. Online. As- sociation for Computational Linguistics.
Ali Alshehri, EI Moatez Billah Nagoudi. and Muham- mad Abdul- MaguleM 2020. Understanding and de- tecting dangerous speech in social media. In The 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT4), LREC.
Yuvanesh Anand, Zach Nussbaum, Brandon Duder- stadt, Benjamin Schmidt, and Andriy Mulyar. 2023. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. hetps: //github. com/ nomic n- ai /ptt4al.
Wissam Antoun, Fady Baly, and Hazem Hajj- 2020. Arabert: Transformer-basee model for arabic lan- guage understanding. In Proceedings of the 4th Work- shop on Source Arabic Corpora and Process- ing Tools, with a Shared Task on Offensive Language Detection, pages 9-15.
/hhups:/lllianneeaa.ca
29 "hutps://arc.. ubc cacube-ar--sookeeee
24 httpss//sstessresssee hww.eesecccccooooootttttaattttte

---

Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Aragpt2: Pre-trained transformer for arabic language generation. In Proceedings of the Sixth Arabic Natu- ral Language Processing Workshop, pages 196-207.
MADAR Arabie dialect corpus and lexicon. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation (I.REC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA).
MS Badawi. 1973. Levels of contemporary arabic in egypt. Cairo: Där al Ma's årif.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165.
Emily M. Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference on Fairness, Acconntability, and Trans- parency, FAceT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. ArXiv, abs/2202.07646.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autore-
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
gressive language modeling with mesh-tensorflow. If you use this software, please cite using these metadata, 58.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman. Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Frrmon, John Etchemendy, Kawin Fiha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas F. Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalturi, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Picch, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R' e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun WU, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Perey Liang. 2021. On the opportunities and risks of foundation models. ArXiv, abs/2108.07221
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220.002111.
Maria De-Arteaga, Alexey Romanov, Hanna Wal- lach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fair- ness, Accountability, and Transparency. FAT '19, page 120-128, New York, NY, USA. Association for Computing Machinery.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilaarco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Ibrahim Abu El- Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul- Abdul-Mageed. 2023. Orca: A challeng- ing benchmark for arabic language understanding.
Claire Cardie Faisal Ladhak, Esin Durmus and Kathleen McKeown. 2020. Wikilingua: A new benchmark dataset for multilingual abstractive summarization. In Findings of EMNLP, 2020.
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dama Abdul- rahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The
Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
12

---

Amelia Glaese. Nathan McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds. Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Cumpbell-Gillnngham, Jonathan Ue- sato, Po-Sen Huang. Ramona Comanescu, Fan Yang, A. See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Ellias, Richard Green, Sovna Mokr'a, a, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William S. Isaac, John F. J. Mellor, Demis Hass- abis. Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dia- logue agents via targeted human judgements. ArXiv, 11s/2209.14375..
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Memory assisted prompt editing to improve GPT-3 after deployment. In ACL 2022 Workshop on Commonsense Representation and Rea- soning.
Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers.
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
Braden Hancock, Antoine Bordes,
Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the As- sociation,e for Computational Linguistics, pages 3667- 3684, Florence, Italy. Association for Computational Linguistics.
Michael McCandless. 2010. Accuracy and performance of google's 's compact language detector. Blog post.
Hamdy Mubarak, Hend Al-Khalifa, and Abdulmohsen Al-Thubaity. 2022. Overview of OSACT5 shared task on Arabic offensive language and hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur' an QA and Fine-Graine Grained Hate Speech Detection, pages 162-166. Marseille. France. European Language Resources Association.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.1555
David M Howeroft, Anja Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg. Sashank San- thanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nig needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Lan- guage Generation, pages 169-182.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language P Processing (Volume 1: Long Papers), pages 5356-55 5371.
El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad A Abdul-Mageed. 2022. AraT5: Text-to- text transformers for Arabic language generation. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 628-647, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Ágata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard. 2019. Way off-policy batch deep reinforcement lear-- ing of implicit human preferences in dialog. ArXiv, abs/1907 1999900000056
Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the- box: An empirical analysis of intersectional occupa- tional biases in popular generative language models. Advances in neural information processing systems, pages 1-224..
El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed. Tariq Alhindi, and Hasan Cavusoglu. 2020. Machine generation and detection arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69 84.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprini 10X.v:2204.0061..
Lorenz Nigst, Maxim Romanov, Sarah Bowen Sa- vant, Masoumeh Scydi, and Peter Verkinderen. 2020. Openiti: a machine-readaal corpus of islamicate texts. http://oii org/10.5281 2281eeoooooo 4075046.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic- I: Technical details and evaluation. White Puper. A121 Labs, I.
OpenAl. 2023. Gpt-4 technical report. ArXiv. abs/2303.08774.
13

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 0rXiv:2110.08207
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Πlię, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli. Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The Ret dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages
Online. Association for Computa- tional Linguistics.
Ethan Perez, Siddharth Karamcheti, Rob Fergus, Ja- son Weston, Douwe Kiela, and Kyunghyun Cho. 2019. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Juint Conference on Natural Language Processing (EMNLP-IJCNL) pages 2402-2411, 2, Hong Kong, China. Association for Computational Linguisties.
Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias in natural language inference. CoRR, 0bs/2105.005541
Olch Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov. Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilin- gual. arXiv preprint arXiv:2204.0755
Adam Poliak, Aparajita Haldar. Rachel Rudinger, J. Ed- ward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018, Collecting diverse nat- ural language inference problems for sentence rep- resentation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 67-81, Brussels, Belgium. Association for Computational Linguistics.
Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. "you are grounded!". Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850-6861.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley. Samyam Rajbhandari. Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arX1v:2201.11990.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog. 1(8).
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 2rXiv:2112.1144:
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy, Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019, Exploring the limits of transfer learning with unified text-to-text trans- former. arXiv preprint arXiv: 1910. 10683.
Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural In- formation Processing Systems, NIPS 20, Red Hook. NY, USA. Curran Associates Inc.
Rachel Rudinger. Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages& 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019, Asynchronous pipeline for process- ing huge corpora on medium to low resource infras- tructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz- Institut für Deutsche Sprache.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine
14

---

Yarden Tal, Inbal Magar, and Roy Schwartz. 2022. Fewer errors, but more stereotypes? the effect of model sive gender bias. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Pro- cessing (GeBNLP), pages 112-120, Seattle, Wash- ingion. Association for Computational Linguistics.
Linting Xue. Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 11934.
Wajdi Zaghouani. Behrang Mohit, Nizar Habash, Os- sama Oheid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followiii llaaa model. https:// github. com/ /tooo//tae tttttttttsuuaaaaaa Lab ford_a
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceed- ings of the 2018 Conference Empirical Methods in Natural Language Processing (EMNLP).
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-TZe Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 2iv: 0.2339.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791 4791-4800, Florence, Italy, Association for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aur'elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Ab- delhak Lakhouaja. 2019. Osian: Open source interna- tional arabic news corpus-prepareet preparation and integration into the clarin clarin-infrastrue infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Work- shop, pages 175-182.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. ArXiv, abs/2212.10560)
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 001066.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint 2XX.:2206600663
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Or- donez, and Kai- Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Short Pupers), pages 15-20, New Orleans, Louisiana, Association for Computational Linguistics.
Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang. Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hen- dricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. 2021. Ethical and so- cial risks of harm from language models. ArXiv, abs/2112.04359.
Wangchunshu Zhou and Ke Xu. 2020. Learning to compare for better training and evaluation of open domain natural language generation models. In AAAI Conference on Artificial Intelligence.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. ArXiv. abs/1909.08599
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. Learning from task de- scriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375. Online. Association for Computational Linguistics.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stien- non, Ryan Lowe, Jan Leike, and Paul Francis Chris- tiano. 2021. Recursively summarizing books with human feedback. ArXiv, abs/2109.10862.
15

---

Appendices
V Examples of Model Output (Appendix D).
In this section, we show examples generated from different JASMINE models under differ- ent settings:
We provide an overview of the Appendix below.
Pretaining data (Appendix A).
Table E.2 shows examples of gener- ated news articles and short stories from JASMINE2.7B under the zero-shot set- ting.
In this section, first provide more details about our JASMINE'S pretraining data. We also give additional details, as follows:
We discuss our decisions about JAS- MINE's vocabulary in Appendix A.I.
Examples from generated 'tweets', prompted from JASMINE are given in Table E.4.
More details on AraC4 Data pro- vided in Appendix A.2.
Table E.5 provides generated 'poctry' from JASMINE2,78, prompted by three lines from Al-Mutanabi (a popular Ara- bic poet) under the zero-shot setting.
The cleaning strategy we employ en- sure the quality of AraC4 presented in Appendix A.3.
Table E.6 shows examples of syntheti- cally generated 'poetry' from our further pretrained JASMIN prompted by a full (or part of) real line of poetry.
II Evaluation Datasets (Appendix B).
We then give more details about the evaluation datasets we created.
We provide full explanation of our AraSwag dataset in Appendix B.1.
Details of poetry dataset are in Ap- pendix B.2.
We provide full details of our speech tran- scription dataset Appendix B.3.
III Evaluation (Appendix C).
We provide additional evaluation details, in- cluding:
Appendix C.I shows an illustrative exam- ple for each word scrambling technique.
The results of the autocompletion datasets (described in 4.4) are in Ap- pendix C.2.
Performance of JASMINEmodels on the NLU tasks is shown in Appendix C.3
IV Analysis Social Bias (Appendix E).
In this section, provide additional informa- tion about social bias analysis.
We provide sample outputs from our so- cial bias analysis in Table E.3.
16

---

A Pertaining data
A.3 AraC4 Cleaning
Table A. shows the distribution of dialect at the country level on AraC4 and Twitter.
For our analysis, we randomly sample 1M para- graphs from AraC4. We first perform language identification using CLD3 (McCandless, 2010) on the data. We find sizable amount of the data (i.e., 13.59%) to be non-Arabic (mostly English or French). We manually inspect ~ 100 random samples of the data predicted as non-Arabic. We find these are mostly either non-linguistic content (e.g., java-script or HTML code) or non-Arabic text. The non-Arabic text is sometimes foreign lan- guage advertising, a full translation of the Arabic text in some cases, even boilerplate text such that in web forums. We clean our AraC4 data by removing HTML tags, elongation, and hash signs. We also reduce repetitive characters, emojis, and emoticons to only two occurrences per instance. Further, we replace URLs with the <URL> string, We finally, keep only webpages that contain at least 95% Arabic characters. We end up with 178GB of Arabic web.
Country
AraC4
Twitter
Algeria
0.48
0.84
Bahrain
2.86
14.82
Egypt
80.48
14.33
Iraq
0.27
1.46
Jordan
0.27
5.19
Kuwait
1.09
13.69
Lebanon
0.32
0.87
Libya
1,85
3,30
Morocco
0.12
0.69
Oman
0.24
4.62
Palestine
1.64
6.25
Qatar
0.36
5.75
Saudi Arabia
0.68
15.12
Sudan
1.42
1.04
Syria
0.07
0.84
Tunisia
1.24
1.73
UAE
0.24
4.50
Yemen
0.08
4.98
B Evaluation Datasets
B.1 AraSwag
Table I: Dialect distribution in percentage on AraC4 and Twitter samples.
Initial Dataset Creation. We randomly sample 10K examples from Arabic WikiHow. We then finetune AraT5 (Nagoudi et al., 2022) on the sam- pled examples separately, where we feed the model with the contexts in order to generate the endings. After finetuning, we generate three possible end- ings for a different set of WikiHow (17K examples). We generate the ending by setting topk 50 and topp = 0.95 to mimic human-like writings. There- fore. our initial datasets contain one context and four endings (one real and three generated).
A.1 JASMINE's Vocabulary
For this, we train the BPE tokenizer on our en- tire dataset. Our choice of vocabulary size is in- spired by Lieber et al. (2021) who demonstrate the benefits of a large vocabulary (e.g., better text representation, faster token processing, and higher ability to cover more content during training and leverage longer prompts in few-shot settings), at the cost of requiring more memory to store the additional parameters of the vocabulary embed- ding layer, as well as more computing resources to calculate the token probabilities using the larger vocabulary. We hence employ a larger vocabulary than GPT-3 (which uses 50K tokens) but choose not to grow it much larger.
Adversarial Dataset Creation. To make the com- monsense inference task more challenging, we fol- low (Zellers et al., 2018, 2019) and apply the adver- sarial filtering (AF) method on the initial dataset. Specifically, on each iteration, the dataset is domly partitioned into Dtrain and Drest with a split of 8:2. We then finetune MARBERT (Abdul- Mageed et al., 2021a) model in order to classify endings as real or generated on D Drvain We evalu- ate the finetuned model on Dtost, then apply AF to replace casy-to-classify generations in Diest with newly generated endings using the finetuned AraT5. This process continues until accuracy of these ad- versaries converges. We observe that during con-
A.2 AraC4 Data
The mC4 dataset Xue et al. (2020) is a multilingual variant of the C4 dataset (Raffel et al., 2019). The mC4 has 101 languages generated from 86 Com- mon Crawl dumps. AraC4, the Arabic part of mC4, represents the 1. 1.66% of mC4 data. It contains 53M webpages with more than 57B Arabic tokens and a total size of 237GB.
hhttps:/wwwwwwwkioow..o
17

---

vergence, the accuracy of MARBERT drops to 30% Finally, we randomly split the resulting AraSWAG dataset into training (Train=11, 288), validation (Dev= 7, 44), and testing (,st=11,655) sets.
Medels
0-shot
1-shot
8-shots
16-shots
24-shots
AraGPTZus
17.82
18.36
21.37
19.59
21.73
ArrGPT2ssi
19.09
20.21
21.34
22.46
24.57
AraGPT25a
21.89
22.29
25.47
26.93
25.35
AraGPT22
22.23
22.56
24.98
25.97
26.33
mGPT.a
12.04
12.27
13,20
14.27
10.41
AraGPTzwr
18.20
19:31
21.70
22.71
25.68
ArGGPT1
22.47
24.26
28.78
JASMINEs
21.64
23.76
25.27
27.43
B.2 Poetry Dataset
JASMINE
22.50 50
22.70
26.01
27.97
98.98
ArGGPT2p
10.72
9.98
9.91
13:21
11:09
The dataset comprises 21.8K Arabic poems from Al-Diwan website which come from 909 au- thors. The poems cover 26 different topies such as romance, politics, religion, etc.
AnGPP22sa
11.34
12.17
14.74
12.57
ArrGPT2y0r
12.44
12.84
13.96
AraGPT2.p
12.81
10.77
12.84
'1.8
12.28
12.95
10.91
T0.42
JASMINRGOM
11.44
11.83
14.32
18.08
13.00
JASMINE,2
14.27
15.03
20.82
21.71
20.81
JASMINEss
15.43
16.65
20.95
23.78
22. 11
JASMINEss
15.57
16.98
19.92
24.84
23.45
B.3 Speech Transcription Dataset
Table C.I: Zero- one-, and few-shot performance the title and paragraph completion tasks.
In order to provide versatile dialectal Arabic dataset that be used evaluate JAS- MINE models' capability to generate dialectal texts, we collect dialectal speech dataset from YouTube. The data come from Arabic soap operas from five different Arab countries. Namely, we collect two soap operas from countries in the set (Algeria, Egypt, Jordan, Morocco, Yemen/. We then manually transcribe 100 utterances, each of length 30 seconds, from each country. We end up with a total of 500 speech utterances from the five different Arabic dialects.
Setting
JASMINEoom
JASMINT,
JASMINES
JASHINES
2:23
1,47
6.34
7:03
8:11
AraNows
7.57
31.12
41.30
44.00
46.
41222
2-dae
0.02
0.27
15
2.20
Aduli
5-dad.
36.71
61.4
0131
$3.20
10-due
17.13
17 20
50
50. 15
24- doct
02 022
45.55
43.02
48
48.66
26.00
37.
0.43
0.31
0.20
1.90
INabet-
3.11
5.09
0.11
2:78
5.85
4.37
5.75
T-deet
0.55
0.35
1.57
Sarcam
S-daa
51.20 225
90.00
90.03
52.55
$4.13
16-due
27.7
43.00
31.20
28.47
16.15
24-dxe
327 30
41.6%
02.19
49.06
57:27
Evaluation Tasks
1.19
2.23
C.1 Words Scrambling
Table C.2: JASMINE evaluation MSA, dialect, and
social meaning text classification tasks (Fi). ). We exclude
The word scrambling task aims to test the mod- cls' ability to correct word-level errors. We use five-word serambling techniques, namely: (1) cy- cle letters, (2) anagramsl, (3) anagrams2, (4) ran- dom insertion, and (5) reversed words. These tech- niques explained in the paper. Table 8 shows illustrative example for each word scrambling technique.
the O-shot setting from NLU results as all the models are not able to predict any correct answers under this setting (i.e., F1=0)
C.3 NLU
We investigate the capability of our models 6 text classification datasets (topic, gender, adult, di- alect, sarcasm, and sentiment) from the ORCA (EI- madany et al., 2023). The performance of JAS- MINE on ARLUE is shown in Table C.2.
C.2 Autocompletion
The autocompletion task aims to predict the last word for a given text. Performance of our JAS- MINE models on news titles, news stories, and the thesis titles datasets are presented in Table C.2.
D Model Output Examples
In this section, we provide various generated ex- amples, including news stories, short stories in Table E.2, social bias Table E.3, tweets in Ta- ble E.4, poetry in Table E.5 and E.6.
26 AI--Dwann website
18

---

E
Analysis of Social Bias
E.1
Social Bias.
In this section, we provide additional information about our social bias analysis. Table E.3 shows generated outputs under different settings presented in appendix E.
E.2 Annotation Guidelines.
For labeling outputs from the model with tags from the set [dangerous, hateful, offensive), two native speakers were given guidelines that include defini- tions for each of the three terms. We provide these definitions here:
Dangerous. Dangerous language pertains state- ments expressing intent to cause physical pain, injury, harm someone for actions taken or not taken. This interpretation does not encompass threats that lack an indication physical harm toward the recipient. Furthermore, this definition excludes instances of playful irony or jest that are intended purely for teasing purposes (Alshehri et al., 2020).
form of retaliation
Offensive. We define offensive language any form of socially unacceptable impolite mate- rial. This encompasses the usage of vulgar lan- guage, profamity, and any explicit or implicit insults or attacks directed towards individuals or groups (Mubarak ct al., 2022).
Hate Speech. Language with hate speech refers to text containing offensive language that targets individuals groups based on shared character- istics, such race (which also includes ethnicity and nationality), religion (inclusive of beliefs), ide- ology (c.g., political or sporting affiliations), dis- ability (covering diseases), social class, and gender (Mubarak et al., 2022).
E.3
List of Professions
Table E. shows the list of 100 occupations we use in our Stereotypical Bias study. The list includes bus driver, lawyer, nurse, etc.
19

---

List of 100 Occupations
الاأتااات ادارة
العتممية الخدمات
البقية الرارد التجارة
المبكررات التممماات
التاارة المطلات
ادارة
الممااات
الطحي الفنوويي
التووي المننل
الأنتااات ادنرة
الباكة
المنية الهندة
المااعم ادارة
العبيية السسررااية
الممارةة الوندمة
التريق
الكحييوي اتطسة ادارة
السمرة
الميييية البندة
الخرلةة
الملومات جي تككراو ادارة
العلب
الالية المناديق أماةة
الملاقة
اليانات قواعد ادارة
البطري العبب
الكميوز مرجة في الععام كصي
التفنية متابية
الكيراافة الاجهوة اا اسلاح
الريافي ا العب
العاعمم
الترريف مااد
الرااسية المدات اصلاح
النسي العلاج
الأداري تليل السوق غميل
التغمي الخاا مالية والراجمة العابة الحاماء
القبي الادارة
بالدليك العلاج
اللبة الادارة
العامة الملاات
العطم تميي
القانوية الاتتاراتت
الأكدمي الممل
البفن الأعداد
الكمياي الحل
الملية الاتتارات
الدفي العمل البيروداي العمل
الباع تطوور
الايية اللزجمات
المدربة الامتمااتت
الالتررزية اا
الراتت تططو
الجررية المرتية
التطوعية الامال
المرجقي المحل
الججيبة تقدي
التناانية المااهة
التممين التحكي الرااني
الباء في المدل
الأهمة تقجبة
الاطفال متامة
الرااني
البيت في الممل
المعما و الااي حراسة
اداري مساعد و البواراتت في الممل الطوارق علب
الححيل
العرطة في الممل
الخغاب من التوصيل خدةة الصملاء خدمة
الللي الحليل
المانع في المصل
الرانني العسريب
الخلغات إراا
الأدمان علاج
العدربس
الغحصية العاابة
الداال خدمة
الكلام اعر علاج
القرحةة
الف
المالاتت سياة الاسنان اا طي
الصياة فني
الليات و الغربي
اللس للأطفال الإجمماعية اا التابية
الميداة ففي
التسوين السبااات ميايك
المكومةة رقاسة طييب مساعد
الارية لاا طبب
الكهرااء فني
الختر فني
النوود في المحل
Table E.1: List of 100 occupations we use in our Stereotypical Bias study.
20

---

News Article
بارااةة إيي الر النع اممات للطايي iسسة موe رربت by الال القيم للر0 الأرردي الأود رنt wid مéلل الغرهي be ورووزز بنن
وفماع والسساد الشمب أمبل من الم عن sain us الجرر life إن سونيوا نوق القيم للكة الأددي الاهد is سعيي A 3ي ملمييي Jás المررفف مير المروين العالين بي وللاسة الرامات قول الم of الملل It الأوديي الالدد مي ونفل الأنمة رامنا ملابي او المفف lis واللن (للl tt d
Original:
يللاب كبي بنية كحر الدي للرااهة وادة مرةة نية حللل بي فني إعن Jem ب االريي مامي مرهة 3ن bes رااال وامدن اللسي البر الاول المببية بوويي بجزاة ني فص JAnل علا فني سشر متب البياد Jبب الولا السسن دار الا el gلمي Jay . المرر Lip الأررر use & wiee suet os iose lay of wal الجامةة الر اا النعب احمات بعع التطاي عنسةة روy مربs io اayy القمم لكر الأرديي الاوير طاس wode John النريي to 1os on الن القمم 8f ف به als علنt of wi + a الأول ابس الريرةة E.s uud of fiv Ao I gede Jay الي الأسما السلل و المية الببة تامة of مsن sلall cis j of الللن الهي y الأرديي alasol Lub wasd الlي gesl July الشمدة الام وزشةة اازررددل الأررية والرية االت الدرو اللوlyy الygيي الربب on dhوe dle aife af woen i jou Jis oll dil net شن الأبن سوورة الدرنة الشربة سحمل ينيا الإححب في الأبن عن 4ee Seth. uuigh الyين of di lean اومان si by jut Ju aie م5 واتا i المفف of & if de'yي qلeتو & le فنم عرري lagheي مeر اiي is ao di giill " ف رجولاا لييا جن الببي الررة رات الي المفف إمنل الارز go 14 il مiو و الااوي اual ofk y ow U zae the is of ay y Jhg ممي المين 1or go An Glols diنll Gpunds معs of deعبl billy i Eouniage of الll cols 1دf i Wae Mun jtf yن العمن الممون من واتلل يامشرطا الغنا of 2000 your gile boggll in اللl اiall wall whd مكي عن عمةة الدنن من وايةة رحلل 1 إعتاع f dajjle wilt الوية العروي legel unl gي المhل المي end ss الtي دفي إرطا 1 عمزل الدمون دنك المسة اسي المب الملوودت aas bp de in was (lanياl weu')l الll aem 3) pull casio ااأيني ونلويا الدعون م المية أناب م تني بال ro of المرل uy way melie of Ju التوي الب مه القرون رنك فدااع ist اللوس الأبن أز البر بي أن أرر مزنة الي لs بيtt واطوو الني العوية الميعي الجر +ي العا شنن حغبل مني الااب اللونف الغغصس في
Generated:
Original:
of -tall sests Wise Mokk Jtf أن الجبمة الممون من واتتيل وامتتر الغاكك on الاال تننس ااني التوس الأمع البسس السناي الناا اققححة عمن التراين سرطات الإماع غمر سنرا d معك التسيl السر مي R المراات سور ابببت الأرري المية أماا -ل ايي من V e! الرراا من الرقنة ف تاا إن ععن أططية wiد waن is| is لصييي المونف من 45 بن طاية des دلi بت والتيي الاااب مالر بن عنdل دي الدعي ب المبع والشرر واككراء والفولاا الله والبرب والترراات اللااك فيي عاار ناام ابعع منن أاyع محن الطماة اباا ياام دs Le السسم سي ون3 J المطن ياام d وروويي god giue اسامم معن اارياان
Generated:
وللييييا والغاادد الlيl W0yy dis a tn الرياس
Short Arable Stories
on sin call jor we عخنو اMl on tie Jois All in godne dl الsاتع diss un الا bby in in ii dir eabl aie of al Si cave Vil lai wlll his aot of i Ja gatadl اiلll باly e/e or of Watd (ill اel Juiin الري at اal pلll i ااعل المرنيl se of Aon sn gutte (ha aie sely ais je of o a was is ele Sf wothle JeJ' jss lass i Walu الyuو all as of i ااlبs التي all ماا j a Ally as an war di العامي J ja f usty fن Jlly All Je 3l fow d اللبي النرع of gotel wid lankig .)f Lie All foy we y يللlly الsيl of العاو الررع je base is ابن 3e 3لl في المرووين of ب اiيl Aas wie عyte الل o eoe 3ud اll in yeu dl cellit wie and agl sb in by 1a die eاdr thes 3f e1 ميو واسييع البع بن ct من بحع Go sheal فصية io uن St النبع ج/و ill wour wit del wou e at اil will sha حل ومييا رون dss الوية فررق is jus uey Ail di Juy wós eù Jرy dn I sسين الl و o/4 ss J (للر is filll on tMt 3o العدد الويد في Wل اممك عs جار laj- ااي An d الررع اهو wes الل الدي رis s/e ejt الو الوو ail wilh de 3a تن di 8) القبعع عن زاا aae ds الماور ااd ببل 13L, واس
Original:
Generated:
والبر البع السر الأيم أسد را x.= I السوق di العع امبي بuر a بنييي joر el للي of والراك المسر e ف سعع عنن شنيةة الأريي واسسن اللع فااي اللل ف موs من الهل أن غنانن السوق إن الوسول أبس سني si لننيع الع الودة ي جراا جلل بf ششر و الماا مه أبي واتتن to If y سبم الsي غرر المرر ba الويي الل وند كوما امدر أسعد الاهية الوان سكك اللا و شفب ططع 1u6 ااههة الهيا is saj االl iin des العال منو d ووسسيا باقطن الأريي So wلlه العع د/ح المون Vi= pus wlll of الليl a has by اعدوز وكاين انسا المحن تن إرطاد
Original:
واطمال البعع الم الأيم ادد مo I السون إi اللع air jus v cssing jit oie el and of chلlly الممر شبم i محن \ا of كسسنن شلرع ع ععر اع علرا ss v السون 3 my الهلر of (Aill of pa uos was Jf uن اللهت السون سن الخريج ا للمول مابب dl leyn وون المون اللع on of الني we المرز ss will ont Jus الll junt el الموز مابي محن اللع ب 3 مام ونت رسك اللي الساع an to اللع e thy :his عاري ss and y ss Jms aيي v bl حنل عن وااد ااي اللع wاي ' juحر lhe sn you unls Jan't Wath النيي ia اليي يرم Yy الني
Generated:
Table E.2: Examples of generated news articles, and short stories from JASMINE 2.78 under the zero-shot setting. We color the initial prompt with gray.
21

---

Gender
السين مأوون ا زن للل حمنةة دزاية أمارت الشسا من 2r Jr الارا شنا الاا
اللركفة وي التغدون المرونن bلlري Uنd العلرية السلات ادزة
الننيةة اأكبب مل وي الطا الواا ي اليامة شرررن iode اللنن أي سن في راري in Wiنe العنني المريب المتية المحة راسااي العسون الماهون 438 ملررke غابا الخدي اتماد
Gender, Coloe, and Region
السية الزااة Ut في عللااتتتت Iز كررااء سسي غار انيةة اري ابواا ب النده لن سير في الأرر المازز بن سون م سسووة الجلن الروريووة المني La Lnl الأركية النسة ولرااب و الملة الأسكي & انن كاب مموو السوو م5ي الورر Joe 3! of in ufl المرل بي ككم oو بلعونع و مصطين كامنن العااع و doنi loghu sjuss الملا
LAl
وااااع الددن لملل الرر ي eاl dygووn الررو الأاري سنe 1l us to LAL
Religion and Beligiion Groupe
باشوووي MA, السة و السة أمس سه مخونة للال عأووننت in Ltll العيية العاب مواا Diu'' من رق غاا أبي كا السدن و التم ملا J جووون Uife A السنون التببب مزاد الوونن of المااع إبل بن عطولة انسل ot غروون li 1 الووني الخابب الحنني yن منوويوnn مع اللال y مميتt احال y عوروة LFls اللدمي العااي VYAA
Table E.3: Sample outputs from our social bias analysis. We color the initial prompt with gray.
Generated Tweets
WWlL Cx# cb in الهم بن -ess sinد of on شار state lo,lea انسان تططر كاريي + باي بني الببب زي محل بي اايl 1ز .bll صعع فسسي if dt y مال فني wل ان بكن bn رلني El فن wht 5n زنه راموت واععنك ا/
أنعع تسيي المبنن برين حلوة اy المتوية المقنية دامم las Iسسلاmin العااة بايع انتع عرات للب in اني زرن
+Rayan AbdelRahman . مغويي زريي JBL المجينن حي وسر وساي إيلسد زاتتهن ببمامنن التسس المكية ينسف . بننيه yا ف المدكل عع وo العاةة ف ال عم 1ر,2 URLla puil f ااo زرني السمريياا' اللرددد سلاان بن we الهم الليد دن غحي أiوي الممر of o of CM w اليية Layl علتتي مناا أرعد اكني امما بااكي of ا y- - زرهة بيي بن ببجح رحح غردة فل كهن 12 lerKoL ااعمي الم في شن عدم الاجة عنم غردة
Table E.4:
Examples of generated 'tweets', prompted, from JASMINE2.78 under zero-shot. We color the initial prompt with gray.
Original Poetry
Generated Poetry
jon oui راال gamtt Geن ea قال of كياء رحب الأي الدرن سيف you gaog gamr 52 4ا- pt du
for sile دااي y dne كوم فاي لf قياه رار (il العووة سis yo يaly " samn Se it ماا (5
تتتم البب بنرر ui فليت so للمرتت -- linat 36 إن دم والسوف إله شطررت ووg 2 iriis الهند وسووف ززكت تن الشم الأأس داي أحس ووان ** كهم الل خلق أعسس فجا نمم طيي ف الف طيي ** طفر عنع الليي المدو فوت
مقح شنرر uf فليت sn الووت linas 35 uf fon من مخيوط وفس االطم sx طربا finm رفيي -حسروا تقق حلوا is تقن f من ** فدر واللوي موge ف ااييي b اممانن بي للراان و وني as سررة If االد فلي مار زان وصحف توراة ونواح o* ااتلن ركية اأزوان وبيت راعاي دحي تاهب رررا s ترمتت أي الهب بين أدعن تقخسب بماااا فلحورنا * بلعوعم كدد تنسب فن من
mn! تيي y io الباة +* وافطاسس الكزفف تديد عنك تا في
pe yا أرف بوارس لا من جaرط ابس عياا تقس أارست
اعم أjرी في بن تصرفت حربا فاتتي جينا رمت أاا
الريان من الريع مززة ** العاي في طيا الفعب معاي وللسان والد الديع فربب . فياا السيي الفتي واكن بججان بي لسار سليمنن ss فياا سار لو جة مااعب المران من كرن واك حفيت * حن واليل خرااا طبتت
الزبان من الريع مززة a0 السان ف طبيا النسب مماي واللان والد الربم فريب tt فياا الريي التي واكن جمان ش لساز سليمنن vo فنياا سار لو بنة ملاعي البان بلورقق مبيا بيا ** at وار فن ما 13! فن جااباا بلقي مiك خيلا للل no نسةة يي leg لأسعلم وللي
الجمان معل أعرانب عل ** فياا الأشمان تففص غدونا كدااي جا الشهاه من رجون 0 عني الخمس هنن وفد فسرت البان من لفر دااليا " باي في مبيا العرق واتيي أراي بلا وقف بي باعر * is إبه تهر o 4اا
تااف دي عني ررحها أان as مشيصصا في مالت الرع إزا
ألكين باقارسس المبس واا "9 تار بالوو الخبي واناا وللنمان بالعبةة مللبك واا باملوككن الدبس ولا
Table E.5: Examples of generated 'poetry', prompted by three lines from Al-Mutanabi, from JASMINE under zero-shot. We color the initial prompt with gray.
22

---

Generated Poems
(I) Empty prompt
نعوا وارس لاع Is oN uلن الكبي 8 حعررة جيت و غي لt lyon at قيف غرب ألعدوا 4att وامهاا رر الله زج Le xs خرب ممف رحن مطخي بب الااي تااعيين اللحط بممال ** الأفن im الليي جان فصن إببه في النت واك xx aلاtاا والتجيم كللدر فيو حان ألسعاء أنق في مي ** بلاااة حسن الAاار عنمل
Gook I العهام بموب و a* الراه مني تماحل وكف
Se في كنن I رلكن ** عجود أن معiم & ومررا K11 مصم وركء ** الممات من تصس ومااا غنا واتططراا ot تiمناا on فادط عناا البدم ونا ج البواع رين الزبات مرف for عن ونعمني فازري أمل عن الخق زوو ** السلمين عن إبب رب 4يا
الغفا بن
(2) Part of poetry'ss 's line prompt
سسm عندد وااي عحuي وهن ** غع تلي من تياه وحم الأمال تن لرجولد ودنت ** اtع و الفنح بان غالررن قيا أادهع أن في تطميي وا x* الموي طلي عن الفسس تمموي للا الهلال يقتبي بور 3وو *x بككك الديا وتاررت
جالل يا علبه فااي مar ** ممري يا نوم
fr فالري مب فككا تقاسن .* اني من يي من بري ولا الم في مزوب ناي حي فان an بارزا بنت tا ci للعع يي عان
الخبلل كسب في اللل عنل راا ** الههد بi ععا أن wl واادء
(3) One line of poetry prompt
علااااا تيي شاسقة بللان xs الكرام بنتت عيي اسفي ياه الروف زم يسس إ/ا s* ستحا عين تامكت ما Whet واا lوgا شوويي بلششت . لصب دي نوو في المم قهي أعراها من براك تجم a* وجلا الدجي جنح زااي ع Wh ف كقق الكهور obin *n دي طلوط بلمملي والجمم
الزيد تمين تدر موعطاتت جديد وعام معن عام بن جليد من بنقي التم وزز ** بسو الزمات ei الزبان بار مe لن العلام ميل lخوا I قد *= عمر بني 1y3o الكي إبع فريم 4 الرجررد في مار عنم ex ميفطا 1ز1 un زورا & العاك م> in liie 3ن ss critt العاو فتتيس ععم بدييد الأكدة فن الرعب ax selual pize KL 4
Lol: يدد العياة فناا ** شطس الأل الي ججية يا
(4) Two lines of poetry prompt
الغلال وماتي لنان بير ** في الأأصنر المصن أمداء العال حهل في الزه وفااء an عرفا طابي الأجم فداء أم
الل ف يكونه أن فنن ws 36 iيا بككك واg بالمجال تدت أن طاي خلا on layeo سيي أن نحي كبت فاا
الصلال في الحب الوفف in ** الناب النع علي تقت
الزرلل في المريي إامركب ** اeي الإياات جكنني Jt ووا أمن بنم فدددءء ** غريا الفي طلية في ونسيت رللل بسفييا أن اl س an ربهط وأحعود عروون أماديه الألال ونار بالهم بعين ** دم الميع في فليس
الرمال قلي في موة *+ قير في باأأللال واليت
اترالي الروم تلك عيي * البراق العون دبع وجمنت
حالي في السنون أددد on byg تبي من تن زال
(5) Three lines of poetry prompt
واعدر بالب في أممي in flr فاايي ميجح فيد الب أن ااا من البي xs حنيي عن القيد وارء الدام طبب من وررنيي sx بالننبب أمعرتي من يا الكام مي بغوق عتب ** من في صعر البع أنصر عاا لبب البب في عناب من x* لفسي عن أحتي أن المغال إلداء من مني إن xs tratt ططع رالطب قوم تاا الجال في تراص طي xx دواء الغلي in الطب وسوس
المقاب جالكيد في بساقق xs جاالي تكرت في فقاله والماب الأككي في خريا buy العي س زهه وجممنيي
رغاب من المين منا راار ** فلي عاا مار uن
عناب من غjة في رسن * بلقاد الأي الففس من بنن
السغاب غبوع عميي في ونكب + الهدوم وقطرر العدوم تطمي
راغغاب رحنة في بطارقي ** 3ن الليي جياله وأن النبرب كحن ما وسيكنف ** بيي j جن ساعلني
Table E.6: Examples of synthetically generated 'poetry', prompted by a full (or part of) real line of poetry empty m..el prompt from our further pre-trained JAS model. We color the initial prompt with gray.
23

---

Dialect
Generated Text
مماات تطبحوا Las طاع 1i1y الشمان مولاي Vlaا ماي dع للوع اونن عية مواتي مساا بطححل مااعع مماء طحمي (sا بن نول الطاني مواي مواك تصكتي عااة عالوهي بن كبيية اي ولl sace كاليي Iا الl عاينة عن مكون eا oي Is will tن oن حيي i خاي خحييةة و ومكو oou If fir swiy A بن ممايي اله عواطف اليب الطيية الأمورر يسمن الل رب i أمينن فاططتة لn واليد عتر احاا أمينة و5ا9 وللاكك و لن عادي والك ابية مااe يا سلادك بي لهم المد زحصن و ماليه ون الشرق وات توب Bile حوف ماققا جوف تتسر ما حنا اييينف ما راعر مادر ببا من اييا داد بووون الواي in وطارر بروو الهل 4n حاداه او مو الغرق للد غول
Algerian
كون كur فن الونوع يي الككن امحا اليي سح علااسييا أممي الميي يا الف سي با العاع باادع عزنp رراا ريه مار Wباtc ممفي dhine Wi عياه بنحبه رادطا
دمررقي الملليي بارب أبن الهم واأبرة الديا ف ددااا وبسوا وغدله بنعلل الي
امم بن الهبم وأدسوه الإاصين أرحم Iا رحت Joل عي عtلle الطبي المالة الشررة داااي عدام العبب jo بي 4 الشرات زز غلفف بن أصية eate I wii ي وركه و لخله 4 حيت لله مي الاار غير واف bot فديي v 3o الطووي والشرية العيج الريع زرققا أن 1ell ماا in تحس عمااا كحمس ما مlام 4 إرنة بين مني ارايي انولراك عاافة مهي Wيmر lay مونيع d عوية مسمك fóوt علوو ان منا Sol Vy inية ف We b wلl we oن غي ماز: خدط im حين iy رريي dجو us tlan oo yos diوe jes vl -يي I بوي مم اددي كحبن عررييي كعا الي و عند برst us whi 25 jor as I smسي اني is ري ميحه مد vا بfر lyاvووn sgriيw fi ا0dl البي 5 I duis امال تدلم جنسس كمان ااا
Egyptian
الله la y his معات بe مuا كريةة الميون حاي بسسسسل
بههي تاممن ابا الممةية في أدا اناب
بمي مبن war فو Whمبا أي اسس yaي مht مم y لاااا يوسفد leeck و كتر ججباا أاا
النن و الحوها Jts مosي مليي كما
Jordanian
الحلاكك ححد تساملل و تحاملك اني الملل بنت عرقش و الل مoع 3 اill ااle uor all uaن الll غنم أن بوسفة أم فيا عليي انس الي الاجتااية و المدبة خررف ياعي & و الل سطاويا in ااب الواب أولاد عid and و بونله اله
تهف نطلم رحت الأوس مددي من خطمطا in we و الشررر مدول an ماوي a راقف الت و كممد سنة ااككف :ي تلت سوا- تبسط مدان مالك إكي را b عi.d لl و واا 3 ge اللنت السلار رر ما قبل ييو أن الطمم ماحب من تطلب العطمم برمس 4ا الول المرط o ابن jo بعرب بن صااي زكب رح زار و ميسوط تكون أنه الهم دااي عمر او ساا عي duy و وكك so bil غبst do فاكس خام حاولء له ااع اللة عي ماهم ش59ي تن احك اني الأكي تفس من
الوووو غامعن انل تخووو و Sseniing 36 yoirll مرري غادي اني ررتي كيية و مدية 3f ف سما ككون سنوعم داكني ul flom for بيككن Sا v ممع عما تطف بعات حاجة في لاا شنر و عللاووا ui -ي بطسد حلعي عي ماجا the كري بني عاجل و دخبدر
Moroccan
بي 3 من بنططا ون العاك بامرر Vi oe تاا و سحاء اك بن تخف و Jاy زرطt , مراةة ككون خاممااا و روك و شال اله رراة و علدع السلام و اخمكي الدي hid is لللب الملنات المنتت المللات من Luat و 4ياا 4ا ماال و فم بني بن جدد المدهي بي و دااف شسن طاربت بار فن dhn seile و عااااار عاااااو حب لنن dl تاعli الرفت مماي عليي إيو شغط مااك تكمل الحسة اليي وواه خادي ف I درnي :ill y دراي من مزي غايي بidي مبوو غن سطلي النهدو جري بلووة فوية معارا مZر و كحك Wuy مو -ي م4م5 و of دet in في مرت داسي حيت كتقوت كaي Sit عتتو نصدرو بيت في تلتوفف ui و فاا تعووف yo تشسكوو كرمترو بيييا الترريلا حديل من حابة أي If مسامم الماا 3i عونة شربر Iو معاه تندي انندد شنتجعع :Kنg) بيت x اقتم ااج بحن الب بنملو كف بروو ia كنن عسمم طنس بوي البنت ف نن مدماا 4ا فاي في انو تن bld بي قف يا tا
البون وي في سومود الدي أن دوي bas ti up الننلل ان حيت اكي اللممم the بن بي تصسي يا اا بني : 4V الجبني in ls! f.y, wلlll daill (لم تتبر ان طارجج : الر سنن اي I of foum في 3ي سوا يسدم وا fic ين الaي Sسروو 30 e من تني 8/ تسس y you dhat yf pots تسيون طاج
Yemen
برروقة ااموةة Alkakl wnt de ' deه الوكسل ونسم wll ly syocl wid ال3ول paiy wlll المرجم وولاا ااعونين &o 3 Lalk, ررحe 4 eor عااج الل قاسي اطا dijm dlptk y0uب
Table E.7: Examples of synthetically generated Arabic dialects text from STGen using JASMINE 2.7B under scro setting, We color the initial prompt with gray.
24


Reviews:
[{"review_id": "L8oKajKCdI", "reviewer": "Reviewer_KASJ", "summary": "", "questions": "Please look at the above notes", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We appreciate the opportunity to respond to your questions and make a number of clarifications.\n\n**Question 1.** What are the reasons to have 80% of the dialectal data written in Egyptian dialect? Will this bring biases to the model? It's known that most tweets are written in Gulf dialects. Why do we have a different distribution here? \"our classifier tags 80% of the predicted AraC4 dialects as Egyptian, 2.86% as Bahraini\"\n\n**Answer 1.** There seems to be a misunderstanding here since AraC4 and the Twitter dataset are two distinct datastes that we use. The sentence you quote refers only to the AraC4 dataset, which does not involve tweets. Rather, AraC4 represents the Arabic portion of multilingual WebForm dataset mC4 (Xue et al., 2020). In order to characterize AraC4, we run SoTa classifiers on it. The classifiers predict both MSA and country-level dialects. Again, the statement \"our classifier tags 80% of the predicted AraC4 dialects as Egyptian, 2.86% as Bahraini\" pertains specifically to the distribution of dialectal data within AraC4 and not in Twitter. This distribution is detailed in Lines [177-195] and further elaborated in Table A.2 (Appendix A) under the \"AraC4'' column. \nRegarding the Twitter data, we provide related information in Table A.2, specifically under the \"Twitter\" column. The distribution in Table A.2 aligns with your observation. Namely, the Gulf dialects are more prominently represented within the Twitter data, with 59.96%, compared to 14.33% Egyptian dialect. Notably, the Gulf dialects' composition, which amounts to 59.96%, can be broken down as follows: Saudi Arabia (15.12%), Bahrain (14.82%), Kuwait (13.69%), Qatar (5.75%), Oman (4.62%), UAE (4.50%), and Iraq (1.46%). We hope this clarification removes any concerns you have about the data we use to pretrain our models. \n\n\n**Question 2.**  You mention perplexity in Table 2. How did you calculate this for Jasmine, AraGPT, and mGPT? Details are missing.\n\n**Answer 2 .** We would like to clarify that we did provide information about perplexity, noting that perplexity is a widely adopted metric that gauges a language model's effectiveness in predicting a given text. In the Evaluation Strategies section in our paper (Lines 217-223), we explicitly state that we adhere to the evaluation methods outlined in (Brown et al., 2020; Howcroft et al., 2020; Zhang et al., 2022) for the assessment of our Jasmine models and baselines. In addition, we provide the corresponding formula in the same section (Lines 221-222) for calculating perplexity (PPL) for each model.  Moreover, we emphasize that due to the lack of a standard dataset for calculating perplexity on Arabic, we curate a new multi-domain dataset of 6K documents from various sources, spanning MSA, dialect, and CA. We hope this information that we have included already in the paper is sufficient. \n\n**Question 3.** Missing details: \"These are news headlines (5K phrases/sentences), news stories (5K paragraphs), and these titles (5K phrases/sentences). All samples are collected from diverse online sources\"\n\n**Answer 3.** We apologize for not providing sufficient details about these datasets. We obtained these autocompletion datasets from various online websites. We will definitely add all the related details to the paper and also release the datasets. We emphasize that we carefully selected these datasets from outside our pretraining data. \n\n\n\n\n**Question 4.** Details in correct word scrambling are missed. Ex: Did you words with their contexts or not? What is the prompt that was given to the model to correct these errors? A word can be corrected in many different ways, and this depends on the context.\n\n**Answer 4.** Again, we provide details about word scrambling in the paper. We first state that we follow the method introduced in GPT-3 (Radford et al., 2019) to generate a synthetic dataset featuring five scrambling tasks (Line 343-345). We then describe each of these five methods (Lines 346-354). We also offer illustrative examples for each method in Table C.1 (Appendix). In each dataset instance, the input constitutes a scrambled word devoid of context. We then task the model to recover the correct version of the word (Lines 355-356). In essence, this is a type of in-context learning based on providing the model with N pairs each of which is composed of a wrong word and its correct form, followed by a wrong word that we then expect the model to correct having seen these N pairs. We run this experiment under different values of N from the set {0, 1, 8, 16, 24}. We provide details about all our extrinsic evaluations, including these N-shot settings, in Section 2 (Lines 225-223). Regardless, even though we have referred the reader to Radford et al. (2019) and provided these details, we will make sure all the information about word scrambling and how we prompt the models is clear in the paper. \n\n**Question 5.** In commonsense inference, how did you generate the 3 false answers?\n\n**Answer 5.** We are sorry you missed the details as to how we generate the three false answers for our commonsense dataset, although we provided these details in the paper. We explain here. For commonsense inference, we create a new challenging dataset that we refer to as AraSWAG. In order to create AraSWAG, we utilize a high-quality dataset from FaisalLadhak & McKeown (2020). We adopt the adversarial filtering method (AF) in Zellers et al. (2019) to generate three hard endings (all of which are false) for a given context, in addition to the original ending.  We briefly describe the  process of generating these false answers (Lines 315-319) and refer the reader to a fully detailed explanation in the appendix. Kindly see our statement “we provide a full explanation of it in Appendix B.1 and Figure B.1 in the same appendix.” (Lines 317-318).\n\n**Question 6.** What is the explanation that sometimes when we increase the number of shots, the model performance drops (ex: sarcasm from 8-shot to 16-shot)? Is there anything in the settings or the prompts?\n\n**Answer 6.** Although this is probably counter-intuitive, few-shot learning can be sensitive to the order of shots as documented in the literature. This can in some cases mean that increasing the number of shots will not necessarily result in better model performance.  We emphasize that we pointed this behavior in the paper upon discussing results of the Autocompletions task. That is, in in lines 302-305 we state:  “We notice a drop in model performance under the 24-shot setting, perhaps since few-shot learning can be sensitive to the order of the shots (Wei et al., 2021; Brown et al., 2020; Lu et al., 2022).” We add that this behavior can also be related the quality of the shots, the length of the samples provided, etc. We refer to the sources cited here. \n\n\n**Question 7.** Adding SOTA results is very useful in the tables \n\n**Answer 7.** We extensively assessed JASMINE across 23 varied datasets, covering Perplexity (6 datasets), Autocompletion (3 datasets), Commonsense Inference (1 dataset), Word Manipulation (7 datasets), and natural language understanding (6 datasets). Notably, 16 out of these 23 datasets are ones that we newly create in this work. In addition, for the six perplexity datasets, we pick a total of 6k instances (1K from each of 6 domains) from publicly available sources not previously used. We needed to create these new datasets in order to provide comprehensive evaluations of our models in this area our work is pioneering. Since these 16 datasets are new, this also means there are no SOTA systems on them that we could compare to. \nNotably, SOTA comparisons are viable only seven datasets. These are the word manipulation task from QALB (Zaghouani et al., 2014) and the six NLU tasks from ORCA (Elmadany et al., 2023). We emphasize that we will incorporate SOTA results from these works in our camera-ready.\n\n\n\n**Question 8.** Prompts in tweets are very short, ex: \"tweet: who/we/how...\". Judging the quality of the generated outputs in such cases has no meaning.\n\n**Answer 8.** Since tweets themselves are very short in nature, we kept the prompts we used brief. In spite of this shortness, we emphasize that our model has been able to generate sensible sequences that look like tweets as illustrated in Table 7 (between Lines 383-384). We show through human evaluation that outputs from the model under these prompts are indistinguishable by humans from real tweets. More precisely, humans are able to detect `tweets` generated by the model in only 48.53% of the time (Lines 424-426). \nFor more intuition, we perceive the function of the prompt here as similar to the prefix used in text-to-text Transformer models (T5), where the goal is just to alert the model that it should output a sequence of particular nature (a tweet in our case). In summary, we believe that since the use of the prompt was successful in achieving our target goal, shortness of the prompt should not be an issue. \n\n\n**Question 9.** The criterion of the selection of the occupations in Table E.1 is not clear.\n\n**Answer 9.** Our occupation list selection is driven by various factors, such as education, skills, and salary considerations. We also consider occupations that are usual in the Arab-speaking world. For example, the list comprises professions such as bus driver, lawyer, doctor, nurse, barber, teacher, and more, totaling around 100 occupations. This compilation aims to encompass a wide range of categories. We will include these details about our list of occupations in the camera-ready version.\n\n**Question 10.** \"Gender, Color, Region\" section is not clear, and many details are missing. \"Combined with our list of 100 professions, this gives us 1,600 generated sentences of which we keep only 1,000 sentences that contain professions. Finally, we manually classify the generated sequences into one of three categories from the manually prepared set {high-wage, medium-wage, low-wage}\"\n\n**Answer 10.** Regarding bias involving occupations, we note that it is not common in most of the Arab world to find publicly available census data about compensation. In view of the lack of this information, we rely on our best estimation in breaking the occupations down into {high-wage, medium-wage, low-wage}. This is done by two native Arabic speakers, each with a Ph.D. who is also well-travelled with extensive knowledge of the Arab world. The two individuals have 100% agreement on the categorization of the model outputs involving professions after discussion. We will  release outputs from the models that we manually label for bias with tags from the set {high-wage, medium-wage, low-wage}. We will also add details about how we break down professions by compensation in the camera-ready.  \nWe note that the model sometimes outputs sequences that do not contain professions. For this reason, we generate 1,600 sequences and keep only 1,000 that involve professions. These are the 1,000 samples that annotators manually label. \nPlease refer to Table E.3 for examples that illustrate this analysis of bias as to gender, color, and region.\n\n**Question 11.** The model should be tested on more tasks from ORCA benchmark.\n\n**Answer 11.** While our primary focus is on NLG tasks rather than NLU tasks, we conducted experiments on six text classification datasets from the ORCA benchmark, constituting 10% of ORCA. However, we will extend our evaluation to the whole of the 60 datasets of the ORCA benchmark. We will add these evaluations to the paper. We thank the reviewer for the suggestion.\n\n**Question 12.** Where is the novelty in this paper?\n\n**Answer 12.** Although we do not propose new methods, our models and the majority of our datasets (16 out of 23) are totally novel. As we state in the paper (Lines 105-126), our contributions include: (1) introducing four GPT models for Arabic, pretrained on large and diverse datasets (412GB); (2) evaluating JASMINE extensively on 23 different datasets (16 of which are novel); including (3) evaluation of the first Arabic GPT models for bias and toxicity; and (4) we plan to release our models responsibly. In addition, our JASMINE models demonstrate novel and emergent abilities such as writing social posts, poetry, and even sensible conversations (Section_5 and Table_7). We hope that the reviewer will see these as sufficient contributions, involving novel models and evaluation datasets, for the extensive amount of work we are offering in this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Results from the generated models beat other models\n2. Model is tested on a diverse testset\n3. Many details and examples are provided ", "reasons_to_reject": "Generally, there are many missing details, biases, etc. in the data creation and testing.\nNotes:\n\n1. What are the reasons to have 80% of the dialectal data written in Egyptian dialect?\nWill this bring biases to the model?\nIt's known that most tweets are written in Gulf dialects. Why do we have a different distribution here? \n\"our classifier tags 80% of the predicted AraC4 dialects as Egyptian, 2.86% as Bahraini\"\n\n2. You mention perplexity in Table 2. How did you calculate this for Jasmine, AraGPT, and mGPT?\nDetails are missing.\n\n3. Missing details:\n\"These are news headlines (5K phrases/sentences), news stories (5K para281 graphs), and these titles (5K phrases/sentences).\nAll samples are collected from diverse online sources\"\n\n4. Details in correct word scrambling are missed.\nEx: Did you words with their contexts or not?\nWhat is the prompt that was given to the model to correct these errors?\nA word can be corrected in many different ways, and this depends on the context.\n\n5. In commonsense inference, how did you generate the 3 false answers?\n\n6. What is the explanation that sometimes when we increase the number of shots, the model performance drops (ex: sarcasm from 8-shot to 16-shot)?\nIs there anything in the settings or the prompts?\n\n7. Adding SOTA results is very useful in the tables\n\n8. Prompts in tweets are very short, ex: \"tweet: who/we/how...\". Judging the quality of the generated outputs in such cases has no meaning. \n\n9. The criterion of the selection of the occupations in Table E.1 is not clear.\n\n10. \"Gender, Color, Region\" section is not clear, and many details are missing.\n\"Combined with our list of 100 professions, this gives us 1,600 generated sentences of which we keep only 1,000 sentences that contain professions. Finally, we manually classify the generated sequences into one of three categories from the manually prepared set {high-wage, medium-wage, low-wage}\"\n\n11. The model should be tested on more tasks from ORCA benchmark.\n\n12. Where is the novelty in this paper?", "questions_to_authors": "Please look at the above notes", "paper_topic_and_main_contributions": "The paper is about training a GPT model on a diverse set of Arabic data (news, tweets, Wikipedia, books, etc.)\nThe authors tested in on different tasks and they showed that it outperforms other GPT models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I think there are missing references for some Arabic benchmarking testsets that are released soon.\nThis includes references for many of the tasks that are partially used in the evaluation.  ", "Typos_Grammar_Style_and_Presentation_Improvements": "I think the paper is well written. However, it will be useful to proof read it and simplify some difficult words."}, "extra_text": {"Justification_for_Ethical_Concerns": "I am not sure about the copyright issues for the training data (ex: tweets, books, etc.).\nAlso, some experiments related to gender/race/hate speech, etc. are not solid, and they depend on the created prompts. These testsets are not available to verify."}}, {"review_id": "GubGZfUDlK", "reviewer": "Reviewer_EqWq", "summary": "", "questions": "A. Will the developed evaluation benchmarks be made available to others in the community to verify and evaluate similar models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review and valuable comments. \n\n**Questions A.** Will the developed evaluation benchmarks be made available to others in the community to verify and evaluate similar models?\n\n**Answer A.** We emphasize that we will make all the developed evaluation benchmarks available to the community. We thank the reviewer for asking.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work is comprehensive in its description of the training procedures, data sets, and evaluation on different tasks including human evaluations. \n\nFurthermore, the submission also addresses some of the inherent biases and shortcomings of the models.\n\nThe availability of these models and benchmarks will substantially advance research on Arabic generative models.\n", "reasons_to_reject": "The submission is very well written, described and evaluated, and I do not see any reason for rejecting the submission.", "questions_to_authors": "A. Will the developed evaluation benchmarks be made available to others in the community to verify and evaluate similar models?", "paper_topic_and_main_contributions": "The submission presents a suite of decoder transformer models for Arabic trained on curated data from both standard and dialectic Arabic, along with evaluation benchmarks.\n\nCompared to AraGPT2 and mGPT, the new JASMINE models mostly outperform previous models, apart from a small number of instances of the largest AraGPT2 and smallest JASMINE model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6qqAjMiJxL", "reviewer": "Reviewer_W8s1", "summary": "", "questions": "none", "limitations": "", "rating": 4, "confidence": 2, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review and valuable comments. We will fix the font size for Table 6. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well written and very well executed.", "reasons_to_reject": "none", "questions_to_authors": "none", "paper_topic_and_main_contributions": "The paper introduces JASMINE, a collection of Arabic GPT models and evaluation tasks. It is a major contribution to unlocking recent advancements of language modeling and in-context learning beyond the English language.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 6 is too small. Maybe adding a second title row with parameter size would free up some horizontal space."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents JASMINE, a collection of Arabic GPT models trained on a diverse dataset encompassing both standard and dialectic Arabic. The paper includes a comprehensive description of the training procedures, datasets, and evaluation on various tasks, including human evaluations. The JASMINE models generally outperform existing models like AraGPT2 and mGPT, with only a few exceptions. The authors also discuss the inherent biases and shortcomings of the models.

Main Contributions:

The authors introduce the JASMINE models, a suite of decoder transformer models for Arabic, trained on a diverse range of data from both standard and dialectic Arabic.
The paper provides a comprehensive description of the training procedures, datasets used, and evaluation on various tasks, including human evaluations.
The JASMINE models generally outperform existing models like AraGPT2 and mGPT, advancing research on Arabic generative models.
The authors also discuss the inherent biases and shortcomings of the models, providing a candid assessment of their performance.

Reasons for Acceptance:

The paper presents a major contribution to the field of Arabic generative models, advancing recent advancements of language modeling and in-context learning beyond the English language.
The JASMINE models generally outperform existing models and are tested on a diverse test set, demonstrating their robustness.
The comprehensive details and examples provided in the paper enhance its clarity and applicability.
The availability of these models and benchmarks will significantly advance research on Arabic generative models.
Decision: Accept-Main