Dolphin: A Challenging and Diverse Benchmark for Arabic NLG
Authors: El Moatez Billah Nagoudi
AbdelRahim A. Elmadany
Ahmed Oumar El-Shangiti
Muhammad Abdul-Mageed
Venue: EMNLP 2023 Findings (2023)
Keywords: Arabic language
Dialectal Arabic
NLG benchmark.

Abstract: We present Dolphin, a novel benchmark that addresses the need for a natural language generation (NLG) evaluation framework dedicated to the wide collection of Arabic languages and varieties. The proposed benchmark encompasses a broad range of 13 different NLG tasks, including dialogue generation, question answering, machine translation, summarization, among others. Dolphin comprises a substantial corpus of 40 diverse and representative public datasets across 50 test splits, carefully curated to reflect real-world scenarios and the linguistic richness of Arabic. It sets a new standard for evaluating the performance and generalization capabilities of Arabic and multilingual models, promising to enable researchers to push the boundaries of current methodologies. We provide an extensive analysis of Dolphin, highlighting its diversity and identifying gaps in current Arabic NLG research. We also offer a public leaderboard that is both interactive and modular and evaluate several Arabic and multilingual models on our benchmark, allowing us to set strong baselines against which researchers can compare.

DOLPHIN: A Challenging and Diverse Benchmark for Arabic NLG
El Moatez Billah NagoudiS.
AbdelRahim Elmadany
Ahmed Oumar El-Shangiti
Muhammad Abdul-Magedd:A
Deep Learning & Natural Language Processing Group, The University of British Columbia DDppartment of Natural Language Processing & Department of Machine Learning, MBZUAI (moatez. nagoudi, a. elmadany, muhammad. mageed jeubc.ca
Abstract
Dour
Question Answering
We present Dolphin, novel benchmark that ad- dresses the need for a natural language generation (NLG) evaluation framework dedicated the wide collection of Arabic languages and varieties. The proposed benchmark encompasses broad range 13 different NLG tasks, including dialogue gen- eration, question answering, machine translation, summarization, among others. Dolphin comprises substantial corpus of 10 diverse and represen- tative public datasets across 50 test splits, care- fully curated to reflect real-world scenarios and a new standard for evaluating the performance and gen- eralization capabilities Arabic and multilingual models, promising to enable researchers to push the boundaries of current methodologies. We pro- vide an extensive analysis of Dolphin, highlighting its diversity and identifying gaps in current Arabic NLG research. We also offer public leaderboard that is both interactive and modular and evaluate several models on our benchmark, allowing us to set strong baselines against which researchers can compare."
Arabizi 5
An mander
Toxe Rosaniing
Paraphrase
DOLPHIN
USS
Thandatonn anstie
DA
GEC
Code Switching
the linguistic richness of Arabic.
Figure 1: Dolphin task clusters and taxonomy. GEC: grammatical error correction. CA: Classical Arabic. DA: Dialectal Arabic. MSA: Modern Standard Arabie.
ent algorithms, models, and techniques. For NLG, benchmarks define specific criteria and metrics for evaluating performance, allowing for objectively gauging the strengths and limitations of different approaches and encouraging healthy competition. NLG benchmarks can also facilitate reproducibility and promote transparency across different studies, acting as a catalyst for advancement the field.
1 Introduction
Natural language generation (NLG) systems at- tempt to produce coherent, contextually appropri- ate. and linguistically accurate human-like lan- guage. These systems have wide range of ap- plications in everyday life, including recreation, education, health, etc. The recent rise of genera- tive models has transformed these NLG systems, making them more relevant and engaging than be- fore. Crucial to measuring the performance of NLG systems high-quality benchmarks. In particular, they provide standardized frameworks for comparing and quantitatively assessing differ-
Despite of this significance, efforts for devel- oping nuanced NLG benchmarks that can allow us to track and guide performance on particular languages remain limited. For Arabic, a wide col- lection of languages and diverse varieties, there is currently no sizeable benchmark that caters to the needs of the community. In this work, present large benchmark for Arabic, dubbed Dolphin, bridge this gap. Our novel benchmark is carefully curated represent real-world usage Arabic scale. Dolphin covers Classical Arabic (CA), pre- modern standardized form of Arabic used for old
'hups://dolhiinn ai/.
"qual contributions.

---

poetry and religious discourse that continues to be employed for literary expression and oration, Mod- ern Standard Arabic (MSA), a modern descendent CA used in formal settings and in pan-Arab me- dia, dialectal Arabic (DA), such as varieties used in everyday communication in the different Arab countries. Dolphin also encompasses text written in both Arabic and Latin scripts, the latter usually re- ferred to as Arabizi. The benchmark is comprised of 13 different generation tasks based on 40 dif- ferent datasets across 50 test splits, making it by far the largest Arabic NLG benchmark to date and among the largest for any group of languages.
16
Coverage:
Arabic
Spuciii
Hultlingua
Dolphis
GC
Number of oa
CUGE
ARGEN
BangaNNu
A/Bench
NTG
GNLGIndaNIG
LOT
CUSE
Number of datasets
Figure 2: Comparison of the number of datasets and tasks supported by the Arabic (including Dolphin), X- specific, and Multilingual NLG benchmarks.
We build Dolphin top of exclusively pub- lic datasets, adding number of newly developed datasets of our creation. This makes Dolphin acces- sible and easy to use. Our benchmark is accompa- nied by a modular leaderboard with a unified evalu- ation metric, i.e., a Dolphin score. The leaderboard is designed to serve as central hub for tracking and showcasing the performance of NLG systems. It functions as dynamic and transparent platform where users submit their models to compare their results against the state-of-the-art approaches. It also encourages a culture of transparency and detailed model description.
details on target languages, dataset sizes, and the breadth of tasks Dolphin covers in Appendix A. Ta- ble I and Figure 2 offer summary of comparisons between Dolphin and other benchmarks.
Arabic Benchmarks. Sajjad et al. (2020) intro- duce AraBench, a machine translation (MT) eval- uation benchmark consisting of five datasets for dialectal Arabic to English translation. AraOpus- 20 (Nagoudi et al., 2022b) is another MT bench- mark of parallel sentences between Arabic and 20 languages. Nagoudi et al. (2022a) introduce Ar- Gen, an Arabic NLG benchmark composed of 19 datasets covering seven tasks. In comparison, Dol- phin is much larger, composed exclusively pub- lic datasets, and covers more varieties. It is also the only benchmark accompanied by a leaderboard. X-Specific Benchmarks. Liu al. (2021) pro- pose GLGE, a generation benchmark for En- glish covering eight datasets across four tasks. CUGE (Yao et al., 2021) and LOT (Guan et al., 2022) are two Chinese benchmarks that cover both language understanding and generation tasks. BanglaNLG (Bhattacharjee et al., 2023) is a gen- eration benchmark designed for Bangala compris- ing seven datasets across six tasks. Guntara et al. (2020) and Doan et al. (2021) present two MT benchmarks for Bahasa Indonesia and Vietnamese languages, respectively.
Overall, we make the following contributions: (1) We introduce novel benchmark for Arabic NLG that is large, public, diverse, and inclusive. (2) We develop a dynamic leaderboard with rich array of best design principles facilitate the mea- surement of progress the field. (3) We evaluate a wide host of Arabic and multilingual models on our benchmark, offering strong baselines. (4) We analyze our benchmark to identify gaps in exist- ing work, hoping to help guide future directions. The rest of the paper is organized as follows: In Section 2, we provide an overview of related work. Section introduces Dolphin design principles and task clusters. In Section 4, we present evaluations of the pretrained models on Dolphin, and discuss the results we acquire. We conclude in Section 5.
2 Related Works
Existing NLG benchmarks can be classified into three distinct categories: Arabic-specific, X- X-specific (where X refers languages other than Arabic, such as English, Chinese, etc.), and multilingual benchmarks. In this section, we provide a brief overview of each category, highlighting their re- spective characteristics and scope. We offer more
Multi-Lingual NLGG Benchmarks. The gen- cration evaluation and metrics benchmark (GEMw) (Gehrmann et al., 2021) multilingual benchmark environment for NLG. GEM,1 features 18 languages across datasets spanning five tasks. Gehrmann et al. (2022) propose a second version, GEM,,, with a new set of datasets and

---

Category Benchmark
Reference
Task Clusters
Lang Datasets
#Clusters
ADT. CS, DRG, DI, GES, MI, NTG, PPH.
Our work
Ar
40
13
Arabic
BOLPHIN
QA. QG. TRW, TRS, MT
ArBench
Sajjad al. (2020)
AraOPUS-20
Nagoudi al. (20226)
MT
Ar
ARGEN
Nagoudi et al. (2022a)
CS. MT. NTG, PPH. QG. TS, TRS
Ar
13
GNLG
Liu et al. (2021)
DRG, DI, TI, IS
En
4
X.Specitcc
BunglaNI.G
Bhattacharjee al. (2023)
MT, TS. QA, DRG, NTG, CLTS
Bn
CUGE
(2021)
QA.
Bahasa Indonesia
Guntarn (2020)
MT
PhoMT
Doan et al. (2021)
MT
LOT
Guan et al. (2022)
RES, DS
Zh
CLSE
Chuklin et al. (2022)
DRG
Mukiligpuul
GEM,
Gehrmann al. (2021)
DRG, DI, RES, IS,
GEM2g
Gehrmann al. (2022)
QA, QG, RES, SEG, BG
IndieNLG
Kumar (2022)
NTG. TS,
MTG
Chen al. (2022)
SG, QG, NTG, 75
IndoNLG
Cahyawijaya et al. (2021)
TS. (AA, CC, MT
10
Table 1: Comparison of NLG benchmarks proposed in the literature across the different covered task clusters. ADT: Arabic text. diacritization. CS: Code-Switching. DRG: dialogue response generation. DT: data-to-text. GEC:
grammatical error correction. MT: machine translation. NTG: title generation. PPH: paraphrase. QA:
question answering. QG: question generation. RES: reasoning. SLG: slide generation. SMP: text simplification.
TRS: transliteration. TRW: text rewriting. TS: text summarization. TT: table text. CLTS: cross-lingual text
summarization. MC: math computation. DR: document retrieval. DS: discourse structure. CC: chit-chat. BG: biography generation. SG: story generation.
challenging tasks. This new version supports 10 documented datasets in 51 languages. Other multilingual NLG benchmarks include CLSE (Chuklin et al., 2022), IndoNL.G (Cahyawijaya et al., 2021), IndieNLG (Kumar et al., 2022), and MTG (Chen et al., 2022). As Figure shows, compared to these benchmarks, Dolphin is the largest both in terms of the number of tasks and datasets. We now introduce Dolphin.
Public datasets. A major principle in choosing our datasets public accessibility as it enables researchers to train and evaluate models without incurring expenses associated with acquiring pri- vate data. For this reason, all our 40 datasets are publicly available.
Rich linguistic variability. In order to accurately reflect the multifaceted and diverse nature of Ara- bic languages and dialectal varieties, strategi- cally incorporate datasets collated from an array of sources, each corresponding different socio- logical and orthographic traditions. Specifically, we construct Dolphin considering four major vari- ants of Arabic: Arabizi (an unconventional method where Arabic is transcribed in Latin script); Clas- sical Arabic (CA); Dialectal Arabic (DA) from a myriad of cities, countries, and regions; and Mod- Standard Arabic (MSA). The heterogeneous nature of our datasets allows for comprehensive representation of Arabic across wide linguistic nu- ances and orthographic traditions. Refer to Figure for an illustrative depiction of the distribution our datasets over various Arabic varieties for each specific task. Table 2 provides quantitative de- scription of these varieties in Dolphin.
3 Dolphin Benchmark
Our objective is to provide comprehensive and challenging benchmark for natural language gen- eration that enables the assessment of language models and the tracking of progress in Arabic. To attain this objective, we develop Dolphin, consid- ering several design principles that we will now elucidate.
3.1 Design Principles
Wide, diverse coverage. As our goal is to offer a demanding and diverse benchmark, we incorporate many datasets from many tasks as is feasible. This allows comprehensive evaluations of LMs. It also facilitates painting complete picture as possible of the limits of current methods across the different tasks. Reflecting this principle, our bench- mark is large. comprises 40 distinct datasets, covering 13 different task clusters.
Standard evaluation metries. Most generation tasks can be evaluated using traditional automated
metrics such as BLEU (Papineni al., 2002) and ROUGE (Lin, 2004). Both of these metrics evalu- ate the n-gram overlap between a reference text

---

and the generated text. Nevertheless, in many tasks (c.g., question generation, open domain gen- eration, title generation) there are multiple valid ways to produce a given text. In our benchmark, in addition to F,, BLEU, and ROUGE, we use several other evaluation metrics such MaxMatch (M2) (Dahlmeier and Ng. 2012) for grammati- cal error correction, and Character Error Rate (CER) (Morris al., 2004) for diacritization.
3.2.1 Machine Translation
The MT cluster built around three tasks: (1) X -> MSA. In this task, we test the ability of the models to translate from six forcign languages into MSA. We use the UN parallel corpus (Ziemski et al., 2016), dataset covering the six official UN languages (i.e., Arabic, Chinese, English, French, Russian, and Spanish). The UN corpus consists of development and test sets only. 2 For training, we randomly select 50K X-Arabic parallel sentences from the multilingual corpus MultiUN (Eisele and Chen, 2010) where X is a language from the six official languages.
Modular, interactive leaderboard. To support fu- ture research, we develop public leaderboard that enables the evaluation of multilingual and Arabic LMs on Dolphin. Our leaderboard is interactive and provides detailed metadata about the corpora such as size, training, development, and test splits, data sources (e.g., URL., GitHub), and citations to publications. The leaderboard also offers details of language models assessed such as the number of parameters, epochs to conversion, pretraining and finetuning information, etc. We provide screen- shot from our leaderboard in Figure D.1. We now introduce cach of the task clusters in Dolphin.
(2) Arabizi X. The goal of this task is to trans- late from Arabizi dialectal text² into one of two foreign languages French and English. For this, we use Darija (Outchakoucht and Es-Samaali, 2021) and NArabizi (Seddah et al., 2020).
(3) Dialects -> English. For this task, we focus on MT from six Arabic dialects into English using the MDP corpus (Bouamor et al., 2014). MDP numan-translatee collection of 1K sentences in Egyptian, Tunisian, Jordanian, Palestinian, and Syrian Arabic, in addition to English. For training. we use the 10K MSA-English manually translated sentences proposed by Bouamor et al. (2018) under 'zrroshho'' condition."
Task Variety
# Clusters
Datasets
# Test Sets
Arobizi
2
Arabial + MSA
3
CA + CA
1
DA : DA
2
3
DA + MSA
1
3.2.2 Code-Switching
DA + En
1
The purpose of the code-switching (CS) task clus- ter is to translate Arabic dialect text that includes code-switching with a foreign language into that foreign language. For this, we create six new human-written (natural) code-switched parallel test datasets, under two tasks: (1) DIA-FR FR. This consists of 300 code-switched Arabic-French tweets collected from Algerian, Moroccan, and Tunisian Twitter. (2) DIA-EN + EN. This is collected from Egyptian, Jordanian, and Pales- tinian Twitter and consists of 300 code-switched Arabic Ennlish posts. For both of these DIA-FR and DIA-EN tasks, a human translation is per- formed by one native speaker from each dialect with semi-native English/French fluency. For these two tasks, perform experiments under the zero- shot setting. That is,
DA-X- + X
1
Toble MSA
MSA- + MSA
21
X MSA
2
Table 2: Descriptive statistics of the linguistic diversity in Dolphin across the different data splits.
3.2 Task Clusters
Dolphin involves 50 test sets curated from 40 datasets. We arrange Dolphin into 13 task clus- ters, as follows: (1) machine translation, (2) code- switching, (3) text summarisation, (4) news title generation, (5) question answering, (6) question generation, (7) transliteration, (8) paraphrasing, (9) text rewriting, (10) diacritization, (11) data-to-text, (12) dialogue generation, and (13) grammatical error correction. Appendix Table B.2 shows a sum- mary of the data splits across datasets and task clusters in Dolphin. We present each task cluster in Dolphin next.
no actual code-
74K sentences that aligned across all official languages. Arabizi the romanization Arabic script (Darwish, 2013). this task, we investigate the Algerian and Moroccan Arabizi.
'Due to lexical overlap hetween Arabic dialects and MSA, this is not zero-shot in the strict sense of the word.

---

switched training data. Rather, we extract 50K MSA-English and MSA-French sentences from AraOPUS-20 (Nagoudi et al., 2022b) that we use for monolingual training. We then extract 50 pairs from each code-switched dialect pair for develop- ment and test on the 250 remainder sentences.
add (6) DAWQAS, an Arabic Why QA dataset (Is- mail and Nabhan Homsi, 2018) our QA cluster. Multi-choice QA. We also use (7) EX- AMS (Hardalov et al., 2020), a cross-lingual multi-choice QA dataset that covers 26 languages (including Arabic). Since we only have this particular test set for Arabic, we follow Hardalov et al. (2020) in evaluating the models on EXAMS under a zero-shot setting.
3.2.3 Text Summarization
For the text summarization (TS) cluster, we use the following five Arabic and multilingual (includ- ing Arabic) publicly available datasets: (1) Mas- siveSum (Varab and Schluter, 2021), (2) XL- Sum Hasan al. (2021), (3) CrossSum (Bhattachar- jee et al., 2021), ) (4) ANT (Chouigui et al., 2021),
3.2.6 Question Generation
The question generation (QG) cluster involves gen- erating question for given passage (Gehrmann et al., 2021). The model is trained generate sim- ple questions relevant passages along with their answers. For this cluster, we use (passage, answer, and question) triplets from five out the seven QA question datasets described in Section 3.2.5.
and (5) MarSum (Gaanoun et al., 2022).
3.2.4 News Title Generation
The news title generation (NTG) task is about pro- ducing suitable title for a given news article. That is, a title generation model is required to output a short grammatical sequence of words that are ap- propriate for the content of the article. For this, we use two datasets: (1) Arabic NTG (Nagoudi et al., 2022a), and (2) XLSum (Hasan et al., 2021).5
3.2.7 Paraphrase
The main goal of this task is to produce for a given Arabic sentence a paraphrase with the same meaning. For this, we employ the following four datasets: (1) AraPara, multi-domain Arabic para- phrase dataset (Nagoudi et al., 2022a), (2) ASEP, an Arabic SemEval paraphrasing dataset (Cer et al., 2017), (3) Arabic paraphrasing benchmark (APB) (Alian al., 2019), and (4) the Arabic sec- tion of TaPaCo (Scherrer, 2020), multilingual paraphrase corpus.
3.2.5 Question Answering
seven publicly available QA datasets across four tasks. A summary of the QA cluster in Appendix Table B.2. We also provide brief information about each task here. Extractive QA. We use four publicly avail- able QA datasets: (1) The Arabic QA dataset ARCD (Mozannar et al., 2019) and the Ara- bie part of the following three multi-lingual QA test sets: (2) MLQA (Lewis et al., 2019), (3) XQuAD (Artetxe et al., 2020), and (4) Ty- DiQA (Artetxe et al., 2020). For all the extrac- tive QA experiments, we finetune on the GoldP multilingual TyDiQAmai (Artetxe et al., 2020) and evaluate on the test sets listed above. Retrieval QA. For this task, (5) LAReQA (Roy al., 2020), cross- lingual retrieval QA dataset built by converting the extractive QA dataset XQuAD (Artetxe et al., 2020) into a retrieval task XQuAD-R. In our benchmark, we focus on the Arabic part of XQUAD-R (AraQuAD-R).
For the QA cluster,
3.2.8 Transliteration
The task of transliteration (TS) about convert- ing a word or text from one writing system to another while preserving the pronunciation and sound of the original language. We create our TS component using three word-level datasets, as fol- lows: (1) ANETA, an English-Arabic named en- tity transliteration and classification dataset pro- posed by Ameur al. (2019). (2) ATAR (Ta- lafha et al., 2021), a word-level parallel corpus con- taining human translations between Jordanian Ara- bizi and MSA. (3) NETrranlliteration (Merhav and Ash, 2018), a bilingual named entity (person names) transliteration dataset mined from Wikidata for English to each of Arabic, Hebrew, Japanese, Katakana, and Russian.
use
Open-Domain QA. In this task, the goal is to an- swer fact-based questions in natural language. We
'we use the multilingual part for Train and Dev, where Arabic data included, and blind-test the Arabic test split. "We exclude the multi-choice QA EXAMS (Hardalov et al., 2020), the open-domain QA DAWQAS (Ismail and Nabhan Homsi, 2018).
We note that XLS (Hasan et al., 2021) has news articles annotated with summaries and titles. We use pairs of articles and titles to create the title generation data.
8 'A informal variant of Arabic spoken in Jordan

---

Datase
Source
Target
ADT
العاي الطاوة بن e انعل eelt up will العووة Is اللمم ابب u 2g 2:ا2ب
3 ( (عق وات شزو الدع العوارة فز العدا النس ري
ABC
wo Julll las jisro at snal
نااي 4 أماون aنن الميية اله طريد مim سمل واي
DRGGA
Tini thes juwn of مقا dgg wtt
DRGGu
lske of & come Loske aas الووي أي fime
وnll Wit of oي &f Jنin شنل f موي عنول 3 مكن pla ماا
DRG
pir العااا بóي sase
sit of
TAPACO
الطماب عسن تووي 3
AraPam
الع بوو ee uلاe المع good جنن of اسع من أرينن Jo الوااي 1o
الموس زرية uي مااع ot العl
APR
اللي lag car
الأيد زق
APGC
تنن أن اسه من زطب م الوبيب es
MI Dadja
meyans had jrida foch tation zwina biail haka the seport is silent about temporary special messures.
It's nice, this park, when it looks good like that
MT EN to MSA
.esyi الما xا will f ان الععب 3 sx
CS30
Villoy waan jlcattig Sio
Eitter he changed aloy or make good cooting
Table 3: Examples from datasets included in Dolphin
3.2.9 Text Rewriting
Egyptian, and Gulf areas were asked to translate 1K utterance-response pairs from the English open- domain dialogues dataset DailyDialog (Li et al., 2017).
The text rewriting (TR) cluster is about generat- ing a text of the target style while preserving the content of the source input text. The TR cluster contains two tasks: (1) DIA - MSA. This task in- volves converting a text written in an Arabic dialect into MSA. For this, we use DiaL2MSA (Mubarak, 20.8). Dial2MSA is parallel dialectal Arabic cor- pus for converting Egyptian, Maghrebi, Levantine, and Gulf dialects into MSA. (2) Gender Rewriting. We use the Arabic parallel gender corpus (APGC) proposed by Alhafni al. (2022), where the lask is to take a given input sentence written in one gender (e.g., male) to produce a target sentence that has the same meaning but employing the opposite gender
3.2.12 Grammatical Error Correction
The task of grammatical error correction (GEC) is focused on analyzing written text, automatically pinpointing, and rectifying a variety of grammat- ical errors as illustrated by a typical instance of grammatical error correction and its manual rectifi- cation. In this cluster, we use three GEC datasets: (1-2) QALB. We use two datasets extracted from the QALB shared tasks from 2014 (Mohit et al., 2014) and 2015 (Rozovskaya et al., 2015). Both datasets are manually corrected collections of Ara- bic texts originating from online commentaries on Aljazeera articles written by native Arabic speakers (LI), as well as texts produced by learners of Ara- bic as second language (L2). (3) ZAEBUC. A corpus that focuses on bilingual writers presented by Habash and Palfreyman (2022). It matches com- parable texts in different languages written by the same writer on different occasions. The corpus is enhanced by adding multiple layered annotations, including manually corrected versions of the raw text, allowing us to use it for GEC.
(i.c., female).
3.2.10 Diacritization
Arabic text diacritization (ATD) is the computa- tional process of restoring missing diacrities or vowels to the orthographic word or a sequence of words (i.c., a sentence or a whole text). For this task, we use the Arabic diacritization dataset proposed by Fadel et al. (2019).
3.2.11 Dialogue Response Generation
Dialogue response generation (DRG) is human- computer interaction task with the goal of auto- matically producing a human-like response given a dialogue context. In this cluster, we have two tasks: (1) MSA DRG. For this task, we use the Arabic empathetic chatbot (AEC) dataset (Naous et al., 2020). It contains open-domain utterances with their corresponding empathetic responses machine translated from English into MSA. (2) Dialectal DRG. We add the open-domain response genera- tion in Arabic dialects proposed by Naous et al. (2023). Three native translators from the Levantine,
3.2.13 Data2Text
The Data2Text (DT) task involves converting struc- tured data like tables as input into descriptive texts without misrepresenting their contents, while sounding natural in writing (i.c., fluently describing this data output). For the DT task cluster, we use the Arabic subset of the multilingual dataset MD2T proposed by Mille et al. (2020) during the third multilingual surface realization shared task. Table 3 shows examples from each task included in

---

Dolphin. We now introduce our strong baselines exploiting our benchmark.
of epochs, learning-rate of 5e batch size of 16, and sequence length of 512. For all the experiments, we identify the best model on the re- spective development split (Dev) and blind testing on the test split (Test). We methodically evalu- ate each task cluster, ultimately reporting single Dolphin score following e.g.. Wang et al. (2018) and Elmadany et al. (2023). Dolphin score is sim- ply the macro-average of the different scores across task clusters, where each task is weighted equally. Since some of our tasks are reported in metrics where lower numbers are better, we split our metric into Dolphin,, score (for tasks where lower 4 is better [i.e., CERJ), and DolphinH score (for tasks where higher 1 is better [i.e., BLEU, FI, M2, and ROUGEJ). Table 4 presents the results of all pre- trained models on each task cluster of Dolphin independently using the relevant metric.
3.3 Comparative Analysis with ARGEN.
Compared to the previous largest Arabic NLU benchmark, ARGEN (which list in Table 1). Dolphin (Nagoudi et al., 2022a) exhibits several advantages. Specifically, we observe the following: Coverage. Dolphin boasts a significantly larger dataset pool (~3X larger). In terms of the number of datasets, Dolphin comprises 40 datasets com- pared to only 13 datasets in ARGEN. Hence, Dol- phin offers a total of 27 totally new datasets.
Task clusters. Dolphin's 's reach also extends to a wider array of task clusters, cncompassing 13 clus- ters as opposed to ARGEN's ' seven clusters. Dol- phin introduces six novel tasks: Arabic text dia- critization, dialogue response generation, data-to- text conversion, grammatical error correction, text rewriting, and question answering.
Discussion. As Table shows, models dedi- cated to Arabic outperform multilingual models on tasks where higher is better (in Dolphing). We also note that AraT5, the new model we build on top of (Nagoudi al., 2022a), achieves the best DolphinH and Dolphing, 27.82 and 11.67, respectively. It is followed by AraBART with Dolphing of 26.44, where a higher score indicates better performance. Conversely, mT5 achieves a Dolphing of 12.42, which is considered better in the opposite scenario. We also note that 2raT5,2 achieves the best results in 30 individual tasks out of 50, followed by AraBART and mTO, where each one excels in 11 and 8 individual tasks, respec- tively." 10
Availability. Dolphin's is datasets are drawn exclu- sively from publicly available sources, while AR- GEN involves several non-public datasets such as the machine translation datasets introduced by Zbib et al. (2012) and transliteration presented by Song et al. (2014). As such, Dolphin avoids issues AR- GEN suffers from such challenges with (i) public distribution of the data and (ii) ease of evaluation. Interactivity. Dolphin uniquely offers a bench- mark Icaderboard, a feature absent in ARGEN, providing real-time performance tracking and a dynamic evaluation environment.
4 Model Evaluation on Dolphin
In order to establish conducive environment for meaningful comparisons on Dolphin, we offer number of strong baselines for both finetuning and k-shoo settings as described next.
Model Computational Costs. We assess the computational efficiency of the Arabic and multilin- gual models we finetune. Figure shows for each model the total time needed for convergence (un- der our 20 epochs constraint with a patience of 5) and the conversion epoch. AraBART the fastest (2.07 hours), with an average of 10.58 epochs to convergence, followed by mT5, AraT5 mTO, and finally AraT5.
4.1 Finetuned Models
For finetuning, we benchmark five different Ara- bie and multilingual models on Dolphin. These are AraT5 (Nagoudi et al., 2022a), of which we pretrain a new version that we refer to as Ara AraBART (Eddine et al., 2022), mBART (Liu et al., 2020), mT5 (Xue et al., 2020), and mTO (Muen- nighoff al., 2022). More information about these models, including our newly introduced AraT5 is in Appendix C.
"Exeept GEC, where we use a seq length of 024.
(ee investigate why AraTS achieves worst, in spite of being dedicated to Arabic, finding to perform better with 100 epochs and a patience of 20 as Nagoudi et al. (2022a) report.
For all models, we finctune on the training data split (Train) for 20 epochs with an early stopping

---

Cluster
Metric
Test
mT0
mT5
AraBART
AraT5
AraT5v2
Dz-Fi Fr
10
11.92
18.67
12.2342.32
J6. 16
Eg-En + En
7.19
4.38
1.35 h065
2.41 4073
3.22
Jo-En + En
11,37
8,42#
2.0
4,59
6,29=am
Code-Switching
Bleu
MA-Ir Fr
11,940.66
13.6340087
16.144002
10.874005
14,48-0.2
Py-Fa +
5,82#0877
7.84
1.170
2.57
3.67=065
Ye-En -> En
8,59+0077
6.914
2.8
3,8840766
5,88=001
Data2Text
Bleu
MD2T
0,22
0.1740.06
0 0.4740.12
0.04
0.83=0.22
Diacritization
CER
ADT І
1.58
1.64
23.43
2.58
1.36=041
AFC
1.29::0.21
1.14
1.71 20.03
1,3
1,41=0.24
DRGBGY
0.05#40.03
0.06
0..04
0.12
0,32=0002
Dialogue Generation
Bleu
DRGoUL
1.02
0, 0:1000007
0.8:0.33
0. 0.29
0,36=0.12
ERGLEV
0.16
0. 0::108
0.57
0.35
0,48
QALB 2014
65.8 65.8640067
66.45
68 67.00.0
64.924023
70.54=0.16
QALB 2015 (L1)
96.90
66.68 66.68+#0.8
69.31 1.1.5
64.22 #082
70.71 #9.61
GEC
Fos (M²
ZAEBUC
47.33=13.34
46.9 96.90720.87
82.0847.54
75.7842248
84,93=2146
TAPACO
15.43#9064
14.89-00.28
17.9
15.90#0006
18.69=0.26
APB
38.36 -8.14
24.29-113.98
37. 37.66*11.0
20.34
30.18 10:
Paraphrase
Bleu
SemEval
20.49:20.13
20.23
24.52
19.33#000
27.9 27.96*33.03
LAREQAGA
63.58
23.38:11122
45.01
25.45+2265
29. 29.93344775
DAWQSØA
2.52
2.82-40.07
4.17410.30
0,3740045
4.98-0.08
EXAMSQA
42.75 #0.61
23.24
22.54
12.69+00.0 40.40
MKQAQA
30,01 #0.41
32.90
32.42
32.92000
33.11 =0.36
Question Answering
LMQAgA
49.17
45.13
47.24
$1.95
54.44 -1.56
ARCDOA
53.24420.4
51.63
50,26
58. 58.12 1
61,38 -1.38-00.7
TyDiQAqA
76.31 4.0.99
74.99-10.25
73.32
39.55
83,34-0.45
XQUADØA
54.55
47.43
47.33
48.71
57,88=0.04
LAREQAqG
9.04
5.542.99
10,23
8.65
10.07=0.56
Sabic-SQUADag
9.20-40.07
9.01
10.10
8.44
10.76-0011
MI.QAgo
6,04
6.0
7.02
6, 6.12
7.45=00.21
Question Generation
Bleu
ARCDoi
17.73#0..99
17.62
22.79
16.8
21.58
TyDiQAgo
30.22
31.00+40.97
+0.64
22.09
33.64=0.89
XQUADoo
10.04
9.96
10.2740.1
9.21 2:009
10.82=0.12
APGC
90.43
90.47=00.04
88. 98.593
29.8770077
91.19=0007
Text Rewriting
Bleu
DIA2MSARY
10.35+00.88
10.26
12.5740.27
10.534000
14.01
XLSum
21.46-00.44
40.64
26.64 6..40.0444
22.71 1.36
26.88=0.02
CrossSum
21.0
20.29
25 25.89t0009
22. 22.14
26.47=1.02
Summarization
RougeL
MarSum
23.0#0.11
22.57=
26.49
21.71 #0.39
25.727=0.02
MassiveSum
25.57=00.11
22.88
30.0
15.89
23.07
ANTCorp
90.29:0.111
88. $8.84
90.0
86 46.642
91.28
Arabic NTG
19.03#0.34
19.23
22.75=t0.9
19.55
22.27=00118
Title Generation
Bleu
XLSum
6.50
6.51
8.98
7.44
9.64=0.13
CER
ANTAEC
19.21
18.9340.30
18. 48.29440.9
20.7440.17
18.44-0.29
Transliteration
CER
ATAR
16.79
16.68
17.70
31.51
15.20=0.32
Belu
NETThans
55,7
55.02
14.155
51.8920044
57.41 =0.93
Darija
16.95
11.2742.54
46.33
1.29
18.09=22885
NArabizi
11.39-11.4
3.379.0.39
11.12-41.20
6,91 4.001
8,98-11.52
En MSA
23.83
23.68
24. 13 13:113
22.34
28.12
MT
Bleu
17.28
17.74440.08
17.76
15.73
20.51 =0.10
+
Es- MSA
19, 19,9240.7
20.56:0.006 0.06
20.3
17.73
21.74=0.36
Ru MSA
16.9340.67
17.12
3.4640.14
14.1040002
18.29=0082
Dolphing Score
Avg. tasks
12.53
12.42
19.81
19.94
11.67
Dolphing Score
Avg. 1 tasks
26.32
23,88
26.44
22.67
27.82
Table 4: Average of three runs of finetuned Arabic and multilingual models Dolphin test. Dolphin, Score: refers to the macro-average scores of tasks where lower score J. is better. Dolphing Score: refers to the macro-average scores of tasks where higher score t is better.

---

Setting
Few Shot
FFT
Training Time (Houre) Best Epoch
Task
BLOOME
ChaiGPT
Axa
Ara15wa
2.76
11.26
CST
11.56
36.61
38.55
40.88
5.56
CST IMSA-I str)
28.41
26.75
28.61
34,61
36.45
37.95
17.49
AraT5
3.71
Discitizatio...
0.51
1.62
1.42
0.11
0.05
0.05
0.02
14.5
Disloguc Genention
0.38
0.38
0.00
CPC
AISBRT
MT (on-bar)
8.33
12.35
10,07
20.52
23.34
23.74
26.71
10.58
MT (es (as-ar)
6.94
9.51
7.33
16.47
17.45
19.32
21.45
(f-+ar)
6.88
5,76
1,97
15.12
15.57
16.26
19.11
2:55
Pamphris
12,98
10:27
10.55
7.89
9.19
9.60
18,90
Question Answeiing
76.04
62.08
60.69
32,98
54.14
53.67
83.16
2.35
Question Generation
28.76
18.53
18.69
14,48
20.08
18.15
34.34
Text Rewriting
76,67
13.97
12.73
11.59
53.34
62.62
90.75
Tile Genenution
0.99
1.20
1.62
3.24
4,62
4.54
9.30
Irassliteration
0.59
0.42
0.42
0.27
0.24
0.23
0.20
Figure Finetuning time (in hrs) no. epoch. We
report the average of three runs across all tasks.
Table 5: K-shot results with BLOOMZ and ChatGPT, compared to best finetuned model (AraT5 We report CER for diacritization and transliteration, ROUGE for summarization, Fo.s (M-) for GEC, and Fi for QA. All other tasks reported in BLEU, L: lower is better.
that arranged in 13 tasks. Dolphin is designed to facilitate meaningful comparisons and encour- age healthy competition in Arabic. We also pro- vide an interactive Icaderboard with a range useful tools and detailed metadata to help situate future research in rich context of information sharing. Dolphin datasets are all publicly available, which should facilitate the adoption and further development of the benchmark. In the future, we intend to build on top of Dolphin by extending more tasks and Arabic varieties.
4.2 Few-Shot Evaluation.
We also carry out k-shot evaluations of both BLOOMZ" (7.1B) (Muennighoff et al., 2022) and ChatGPT (gpt-3.5-turbo) on 12 different NLG tasks across 16 test sets extracted from Dolphin." 13 To keep the manageable, randomly sample set of 200 examples from the test of each task for evaluation. We then evaluate both models under 0-, 5-, and 10-shot settings. For all experiments, we set the temperature to zemo to generate determin- istic and reproducible results. We compare both models' performance to our best fully finetuned model, 2r, blind-tested on the same sampled 200 examples.
Limitations
In spite of the diversity, wide-coverage, high- quality datasets, accessibility, and challenging ture of Dolphin, it is not without limitations. In particular, we identify the following limitations.
1. Coverage of Arabic Varieties. While make efforts incorporate tasks from all Ara- bic varieties, it is important to note that there a lack of available downstream datasets from countries such as Djibouti, Mauritania, and Yemen. Consequently, these varieties are not currently included in Dolphin. We hope that the community will develop re- sources representing all Arab countries, in- cluding these, across the various tasks. We also hope that future versions of our bench- mark will have extended dialectal coverage in ways that enhance its representation of the Arabic language and help foster technological inclusion.
Discussion. Tables 5, shows that ChatGPT out- performs BLOOMZ in all the NLG tasks under
0,, 5-, and 10-shot settings. The only exception is the text rewriting task the 0-shot setting. It is worth mentioning that Ara outperforms both ChatGPT and BLOOMZ by 14 out of 16. How- ever, ChatGPT (/0-shot) achieves the highest score in both code-switching tasks, perhaps due to its multilingual pretraining data.
5 Conclusion
We presented Dolphin, large and diverse bench- mark for Arabic NLG composed of 40 datasets
BLOOMZ is finetuned multiple tasks in 46 languages, including 1% Arabic.
We evaluate version existing on March 1st, 2023. We only exclude the data-to-text task.
Machine-Translater
Datasets. Dol-
machine-translated phin includes two machine translated

---

data, ABC (Naous et al., 2021) and Ara- Para (Nagoudi et al., 2022a)). While these datasets increase task coverage Dolphin, the MT process may inadvertently introduce some biases For example, MT can result in a narrow representation of language patterns and structures, leading to a limited under- standing of the complexities and nuances of different languages. Additionally, benchmark datasets may not adequately capture the wide range of domains, genres, and styles that exist in real-world translation scenarios. This can limit the generalizability of models trained on such data, as they may struggle to handle unfamiliar or specialized content. We hope that future versions of Dolphin will involve real-world data that further complement (or even substitute) these translated datasets.
leaderboard website.
1. Data Privacy. Regarding data involved in Dolphin, we develop the benchmark using publicly available data. For this reason, we do not have significant privacy concerns. addi- tion, the new datasets we develop and release for code-switched machine translation have undergone manual inspection to ensure there is no unintended leak of privacy information in any of the samples.
2. Intended Use. We believe our work will spur further research on studying language models Arabic NLG benchmark. We create publicly available leaderboard and benchmark several multilingual and Arabic- dedicated SOTA models on Dolphin. The benchmark will facilitate a unified evaluation and pave the way for a healthy competition that could push SoTA on Arabic language gen- eration.
3. Automated Evaluation. Although all NLP depends heavily on automated evaluation speed up model development, automated methods have their limitations, especially for some tasks. That is, in addition to automated evaluation, some tasks may need human eval- uation. In particular, we believe human eval- uation can play crucial role in NLG tasks such as open-domain dialogue generation. For example, it can capture the nuanced aspects of dialogue quality, such coherence, rele- vance, and appropriateness. In addition, hu- man evaluation can allow for a comprehen- sive assessment of the generated dialogues, taking into account contextual understanding, fluency, and overall user experience. This feedback is invaluable in refining and improv- ing dialogue generation models, ensuring that they meet the high standards of human-like conversation.
3. Potential Misuse and Bias. The datasets we collect to create Dolphin may contain poten- tial harmful contents. Additionally, the mod- els we evaluate might be exposed to bias and result may generate unintended contents. Therefore, we recommend that these datasets and models not be used in applications with- out careful prior consideration of potential misuse and bias.
Acknowledgements
We gratefully acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; P0T7N-2018-026611 the Social Sciences and Hu- manities Research Council of Canada (SSHRC;
435-2018-0576; 895-2020-1004; 895-2021-1008),
7 Ethics Statement
Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada," 14 UBC ARC- Sockeye.' 15 We thank the Google TFRC program for providing with free TPU access." 16
Data Collection and Release. Dolphin is based on publicly available datasets that would not be pos- sible without the hard work of a large number of researchers over the years. We are grateful for these efforts invested by pioneer colleagues. One down- side of benchmarking could be that the original authors of the different datasets are not sufficiently acknowledged. In our work, we make sure that all publications of resources we use are properly cited, both by referencing these in this paper (Sec- tion 3) and highlighting them in our GitHub and
References
Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. User-Centrr Gender Rewriting, In Proceed- ings of the 2022 Conference of the North American
"hups://llianceean.ce
https://arrc ubc calubc-arc-sockeyy
https:///sstsss. www.tttssssssrrch.ccoo www.aach..ooglerrrraaaaeoee

---

Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 618- 631, Seattle, United States. Association for Compu- tational Linguistics.
Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang, Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li. 2022. MTG: A benchmark suite for multilin- gual text generation. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 2508-2521 Seattle, United States. Association for Computational Linguistics.
Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building arabic para- phrasing benchmark. In Proceedings of the Second International conference on Data Science E-learning and Information Systems (DATA' 2019), pages 1-5.
Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2021. An arabic multi-source news corpus: Experimenting on single-document extractive sum- marization. Arabian Journal for Science and Engi- neering, 46:3925-3938.
Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2019. Anetac: Arabic named entity transliteration and classification dataset. arXiv preprint 1907.0310
Aleksandr Chuklin, Justin Zhao, and Mihir Kale. 2022. Clse; Corpus of linguistically significant entities. pages 78-90 96.
Wissam Antoun, Fady Baly, and Hazem Hajj- 2020. Arabert: Transformer-basee model for arabic lan- guage understanding. In Proceedings of the 4th Work- shop on Open-Source Arabic Corpora and Process- ing Tools, with Shared Task on Offensive Language Detection, pages 9-15.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 568-572.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transfcrability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637.
Kareem Darwish. 2013. Arabizi detection and conver- sion to arabic. arXiv preprint arXiv:1306.6755.
Long Doan, Linh The Nguyen, Nguyen Luong Tran,
Abhik Bhattacharjec. Tahmid Hasan, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong bin Kang, and Rifat Shahri- yar. 2021, Crosssum: Beyond english-centric cross- lingual abstractive text summarization for 1500+ lan- guage pairs CoRR, abs/21 12.08804.
Thai Hoang, and Dat Quoc Nguyen. 2021. PhoMT: A augl-tualtty and large-scale benchmark dataset for Vietnamese- Engeess machine translation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4495-4503, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, and Rifat Shahriyar. 2023. BanglaNLG and BanglaTS: Benchmarks and resources for evaluating low-resource natural language generation in Bangla. pages 726-735.
Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. Arabart: a pretrained arabic sequence-to-sequeece model for abstractive summarization.
Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In LREC, pages 1240-1245.
Andreas Eisele and Yu Chen. 2010. MultiUN: A mul- tilingual corpus from united nation documents. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC' 10), Valletta, Malta. European Language Resources Asso- ciation (ELRA).
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obcid, Salam Khalifa, Fadhl Eryani, Alexan- der Erdmann, al. 2018. The Madar Arabic Dialect Corpus and Lexicon. In Proceedings of the Eleventh International Conference on Language Resourees and Evaluation (LREC-2015)
RbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad
2023, ORCA: A Chal- lenging Benchmark for Arabic Language Understand- ing.
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kun- coro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Leylia Khodra, et al. 2021. Indonlg: Bench- mark and resources for evaluating indonesian natural language generation. pages 8875-8898.
Mohamed Elmahdy, Mark Hasegawa-Johnson, and Eiman Mustafawi. 2014. Development of TV broadcasts speech recognition system for qatari Ara- bic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 3057-3061, Reykjavik, Iceland. European Language Resources Association (ELRA).
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarit- and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
Ali Fadel, Ibraheem Tuffaha, Bara' Al-Jawarnch, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritiza- tion using deep neural networks.

---

Kamel Gaanoun, Abdou Naira, Anass Allak, and Imade Benelallam. 2022. Automatic Text Summarization for Moroccan Arabic Dialect Using Artificial In- telligence Approach, pages 158-177.
Tahmid Hasan, Abhik Bhattacharjee. Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Bin Kang, M. So- hel Rahman, and Rifat Shahriyar. 2021. XI-sum: Large-scale multilingual abstractive summarization for 44 languages.
Sebastian Gehrmann, Tosin Adewumi. Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan. Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan- Majjilan-Maajo, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan. Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perrz-Beltrachini. Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez. Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt. Nishant Subramani, Wei Xu, Diyi Yang. Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. pages 96-120.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross goos--lngual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Walaa Ismail and Masun Nabhan Homsi. 2018. Dawqas: A dataset for arabic why question answering system. Procedia Computer Science,
Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Ku- mar. 2022. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages $363-53944 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Demoising srgueecce-t--equenee pre-training for natural language generation, translation, and com- prehension. pages 7871-7880.
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Memillan-major, Anna Shvets, Ashish Upadhyay, and Bernd Bohnet. 2022. GEMv2: Multilingual NLG benchmarking in a single line of code. pages 266-281.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Eval- uating cross-lingual extractive question answering. arXiv preprint arx0v:1910.0747..
Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang, 2022. LOT: Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation. Transac- tions of the Association for Computational Linguis- tics, 10:434-451.
Yanran Li. Hui Su, Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ( (Volume 1: Long Papers).
Tri Wahyu Guntara, Alham Fikri Aji. and Radityo Eko Prasojo. 2020. Benchmarking multidomain English- Indonesian machine translation. In Proceedings of the 13th Workshop on Building and Using Compara- ble Corpora. pages 35-43, Marseille, France. Euro- pean Language Resources Association.
ural Language Processing pages 986-995. Taipei, Taiwan. Asian Federation of Natural Language Processing.
Chin- Yew Lin. 2004. Rouge: A package for auto- matic evaluation of summaries. Text Summarization Branches Out.
Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic- English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79-88, Marseille, France. European Language Resources Association.
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang. Jiusheng Chen, Daxin Jiang. Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, and Nan Duan. 2021. GLGE: A new general language generation evaluation benchmark. pages 408-420.
Momehil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslay Nakov. 2020. EXAMS; A multi-subject high school examinations dataset for cross-lingus and multilin- gual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- On-
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Tranxactions
guage Processing (EMNLP), pages line. Association for Computational Linguistics.

---

of the Association for Computational Linguisties, 8:726-742.
Tarek Naous, Wissam Antoun, Reem Mahmoud, and Hazem Hajj. 2021. Empathetic BERT2BERT conver- sational model: Learning Arabic language generation with little data. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 164- 172, Kyiv, Ukraine (Virtual). Association for Compu- tational Linguisties.
Yuval Merhav and Stephen Ash. 2018. Design Chal- lenges in Named Entity Transliteration. In Proceed- ings of the 27th International Conference on Compu- tational Linguistics, pages 630-640, Santa Fe, New Mexico, USA. Association for Computational Lin- guistics.
Tarek Naous, Zahraa Bassyouni, Bassel Mousi, Hazem Hajj, Wassim El Hajj, and Khaled Shaban, 2023. Open-domain response generation in low-resource settings using self-supervised pre-training of warm- started transformers. ACM Transactions on Asian and Low-Resource Language Information Process- ing, 22(4):1-12.
Simon Mille, Anya Belz, Bernd Bohnet, Thiago Cas- tro Ferreira, Yvette Graham, and Leo Wanner. 2020. The third multilingual surface realisation shared task (SR'20): Overview and evaluation results. In Pro- ceedings of the Third Workshop on Multilingual Sur- face Realisation, pages 1-20, Barcelona, Spain (On- line). Association for Computational Linguistics.
Tarek Naous, Christian Hokayem, and Hazem Hajj- 2020. Empathy- arabic conversational chatbot. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 58-68.
Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wa- jdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39-47, Doha, Qatar. Association for Computational Linguistics.
Aissam Outchakoucht and Hamza Es-Samaali. 2021. Moroccan dialect -darija- open dataset.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting association for compu- tational linguistics, pages 1-318, Association for Computational Linguistics.
Andrew Morris, Viktoria Maier, and Phil Green, 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition.
Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. arXiv preprint arXiv: 1906.05394.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019, Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv preprint arXiv: 10683.
Hamdy Mubarak. 2018 Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. In OSACT 3: The 3rd Workshop Open-Source Arabic Corpora and Processing Tools, page 49.
Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Language-agnosttc answer retrieval from a multilin- gual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5919-5930, Online. Association for Computational Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Seao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al- banie, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning.
Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wa- jdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015. The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Pro- cessing, pages 26-35, Beijing, China. Association for Computational Linguistics.
El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed 2022a. AraT5:: Text- to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Hassan Sajjad, Ahmed Abdelali, Nadir Durrami, and Fahim Dalvi, 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094-5107.
1: Long Pupers), pages 628-647, Dublin, Ireland. Association for Computational Linguistics.
Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul- MaduulMageed. 2022b. TURJUMAN: A public toolkit for neural Arabic machine transla- tion. Proceedinsg of the Sth Workshop on Open- Source Arabic Corpora and Processing Tools with Shared Tasks Qur' an QA and Fine-Grained Hate Speech Detection, pages 1-11. Marseille, France. Eu- ropean Language Resources Association.
Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yves Scherrer. 2020. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfft Language Resources and Evaluation Confer- ence, pages 6868-6873, Marseille, France. European Language Resources Association.

---

Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoit Sagot, and Abhishek Srivastava. 2020. Build- ing a user-generated content North-African Arabizi treebank: Tackling hell. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 119--1550 Online. Association for Computational Linguistics.
v1. O. In Proceedings of the Tenth International Conference on Language Resources and Evaluation 16), pages 3530-3534.
Zhiyi Song, Stephanie M Strassel, Haejoong Lee, Kevin Walker, Jonathan Wright, Jennifer Garland, Dana Fore, Brian Gainor, Preston Cabe, Thomas Thomas, et al. 2014. Collecting natural sms and chat conversa- tions in multiple languages: The bolt phase corpus. LREC, pages 1699-1704. Citeseer.
Bashar Talafha, Analle Abuammar, and Mahmoud Al- Ayyoub. 2021. Atar: Attention based Istm for arabizi transliteration. International Journal of Electrical and Computer Engineering, 1::322-2334.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces OPUS. pages 2214-2218.
Daniel Varab and Natalic Schluter. 2021. Mas- siveSumm: a very large-scale, very multilingual, summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150-10161, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyz- ing and Interpreting Neural Networks for NLP, pages 353-355, Brussels. Belgium. Association for Com- putational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXh::2010.11933
Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fan- chao Qi, Junwei Bao, Jinran Nie, et al. 2021. CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark. arXiv e-prints, pages arXiv- 2112.
Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stal- lard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burchh 2012. Machine translation of Arabic dialects. In Pro- ceedings of the 2012 conference the north amer- ican chapter of the association for computational linguistics: Human language technologies, pages 49-59.
Michal Ziemski, Marcin Junczys- Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus

---

Appendices
corpus (Blmahdy et al., 2014), and the English Bible translated into MSA, Tunisian, and Mo- rocco."?
We organize our appendices as follows:
AraOPUS-20 This is an MT benchmark proposed by Nagoudi et al. (2022b). It consists of parallel bitext between Arabic and 20 languages extracted from the OPUS publicly available corpora (Tiede- mann, 2012). The languages paired with Arabic include high-resource languages such as English, French, and Spanish and low-resource ones such Cehuano," Tamashek, 19 and Yoruba." 20
Sections list:
NLG Benchmarks. (Section A)
Arabic (Subsection A.I)
X- X-Specific (Subsection A.2)
Multi-Lingual (Subsection A.3)
Dolphin Tasks (Section B)
ARGEN. The Rabiic natural language GENeration (ARGEN) benchmark was in- troduced by Nagoudi et al. (2022a). is composed of 19 datasets and covers the seven tasks: machine translation, code-switched text translation, summa- rization, news title generation, question generation, paraphrasing, and transliteration.
S2S LLMs (Section C)
Public Leaderboard. (Section D
Tables and Figures List:
Statistics of our Dolphin benchmark across the different task clusters (Table B.2).
A.2 X-Specific Benchmarks
Dolphin's 8 Leaderboard (Figure D.I)
GLGE. The General Language Generation Evaluation(GLGG) by Liu et al. (2021) is a multi-task benchmark for evaluating the general- ization capabilities of NLG in the English lan- guage. GLGE has eight English language gener- ation datasets, covering four NLG tasks: data-to- text, dialog, table-to-text, and summarization. BanglaNLG. BanglaNLG a benchmark designed for Bangala Bhattacharjee ing seven datasets across six NLG tasks: machine translation, text summarization, question answer- ing, dialogue generation, headline generation, and cross-lingual summarization.
A NLG Benchmarks
Existing NLG benchmarks can be classified into three distinct categories: Arabic-specific, X-specific (where X refers to languages other than Arabic, such as English, Chinese, and others), and multilin- gual benchmarks. In this section, shall provide a brief overview of each category, highlighting their respective characteristies and scope. We will high- light aspects such as the target language, dataset size, and the breadth tasks covered. This analysis is summarized in Table and Figure 2. The current NLG benchmarks can be divided into three main groups: benchmarks that focus on Arabic, bench- marks that focus languages other than Arabic (X-specific), and benchmarks that cover multiple languages. In this section, we will give a brief sum- mary each category, emphasizing their unique features and scope. We will discuss factors like the target language, dataset size, and the range of tasks included.
(2023) compris-
CUGE. The Chinese Language Understanding Generation Evaluation Benchmark Yao et al. (2021) covers both language understanding and generation. The language generation collection contains nine datasets across eight tasks. The tasks are open-domain question answering, document retrieval, summarization. data-to-text, knowledge- driven conversation, machine translation, cross- lingual text summarization, and mathematical com- putation. The benchmark also covers the tasks of grammatical error correction and reverse dictionary generation, but treats these under the NLU compo- nent.
A.1 Arabic Benchmarks
AraBench. AraBench is an evaluation benchmark for dialectal Arabic to English machine transla- tion (MT) introduced by (Sajjad al., 2020). It consists of five publicly available datasets: Arabic- Dialect/English Parallel Text (APT) (Zbib al.. 2012), Multi-dialectal Parallel Corpus of Arabic (MDC) (Bouamor et al., 2014), MADAR Cor- pus (Bouamor et al., 2018), Qatari-Engliss speech
L'The United Bible Societies http://www.bbh http:///www.bbble.om com
LLanguage spoken in the southern Philippines "Tnnashe variety of Tuareg. Berber macro- language widely spoken by nomadic tribes across North Africa coun- tries.
Yoruba is a language spoken in West Africa, primarily in Southwestern Nigeria.

---

Bahasa Indonesia. The Bahasa Indonesia lan- guage has over 200M active speakers, yet still considered a low-resource language. To overcome this problem, (Guntara et al., 2020) introduced a machine translation benchmark with 14 datasets across four domains: news, religion, conversation, and general.
for datasets and models, with an online evaluation process that collects model oulputs and computes metrics for all datasets. GEM,2 is built around nine NLG tasks tata-to-text, dialog response gener- ation, paraphrasing, generative question answering, question generation, reasoning, slide generation, simplification, and summarization.
PhoMT. Doan al. (2021) introduces new Vietnamese-Engglshh parallel dataset that larger and of higher quality than the existing benchmark corpus. The authors conduct experiments to evalu- ate various translation models on the new dataset and find that the best performance is achieved by fine-tuning the pre-traie rre-trained sequence-to peuuence-o--equenee sequence denoising auto-encoder mBART.
IndicNLG. The first benchmark for Indic lan- guages Kumar et al. (2022) covers 11 Indic lan- guages belonging to two language families: Indo- Aryan and Dravidian. IndicNI.G involves the five following tasks: biography generation, news headline generation, sentence summarization, para- phrase generation, and question generation.
MTG. Chen et al. (2022) introduce the Multilingual Text Generation to promote knowledge transfer and cross -oosull generation between arbitrary language pairs. MTG contains 400K of humanly annotated data samples in five languages, covering four generation tasks. These are story generation, question generation, title generation, and text summarization.
LOT. The LOng Text understanding and genera- tion benchmark targets Chinese long text model- ing in a story-centric manner Guan al. (2022). LOT combines two comprehension tasks and two- generation tasks. The two generation tasks are commonsense reasoning and discourse structure.
A.3 Multi-Lingual NLG Benchmarks
B Dolphin Tasks
IndoNLG. IndoNLG covers three low resources languages widely spoken in Indonesia: Indonesian, Javanese, and Sundanese Cahyawijaya et al. (2021). It consists of ten distinct datasets, encompassing four tasks. These are summarization, question an- swering, chit-chat, and machine translation. CLSE. The Corpus of Linguistically Significant Entities Chuklin et al. (2022) is a multilingual named entities corpus that covers 34 languages, 74 semantic classes, and 222 distinguishable lin- guistic signatures. The authors also developed an expanded version of the Schema-Guiee Guided Dialog Dataset (SG-CLSE) illustrate one of the potential uses of CLSE in three languages: French, Marathi, and Russian.
C Arabic and Multilingual S2S LLMs
In this section, we list the Arabic and multilingual Sequence-to-sequeee (S2S) pretrained LMs we finetune on Dolphin.
AraT5. (Nagoudi et al., 2022a) is an adaptation of the T5 model specifically designed for the Arabic language.
pre-trained on large (248GB of Arabic text) diverse (MSA and Arabic dialects) dataset to effectively handle different Arabic tasks. In addition to Arabic, AraT5's vocabulary covers 11 other languages. In this work, we evaluate new in-house version of AraT5 dubbed AraT5,2. AraT5, Our analysis shows that Ara'S5 requires large number of epochs converge, making it an expensive model. For this reason, we pretrain a new version of the model from scratch exploiting a larger (~ 400GB) and more diverse pretraining dataset than used by (Nagoudi et al., 2022a). As we show in our results, the new model converges faster than AraT5 and achieves better results under our cap of 20 epochs for finetuning across all models. AraBART. (Eddine et al., 2022) is model based on the encoder-decoder BART base archi- tecture (Lewis et al., 2020), featuring six encoder and 6 decoder layers. is pretrained on the same corpus as AraBERT (Antoun et al., 2020), with reversed preprocessing for more natural text gener-
GEM,1- The Generation Evaluation and Metrics benchmark (Gehrmann et al., 2021) is multi- lingual benchmark environment for NLG. GEM features 18 languages across 13 datasets spanning five NLG tasks: data-to-text, dialog response gen- eration, reasoning, summarization, and simplifica- tion.21 21
GEM,2. Gehrmann et al. (2022) propose a second version, GEM,2, styled after GEM,1 with a new set of datasets and more challenging tasks. This version supports 40 documented datasets in 51 languages. It introduces modular infrastructure
Two of the datasets do not include English at all.

---

Task Cluster
Task
Test Set
Source
Train"
Dev
Test
De + Ar
Eisele and Chen (2010))
50K
4K
4K
Fn - Ar
50K
4K
4K
> MSA
Fr Ar
Ziemski al. (2016)
50K
4K
4K
Ru Ar
50K
4K
4K
Dr Fr
Seddah al. (2020)
11K
144
146
MT
Arabizi - X
Mu
Outchakoucht and Es-Samaali (2021)
8K
Eg
200
200
Nagoudi et al. (2022b)*
DA + En
+ En
Bouamor al. (2014)13
50K
200
Sy - En
200
800
Tn + Ein
200
800
Dz-Fr Fr
50
250
Ma-Fr
50
250
Tn-Fi Fy
Nagoudi et (2022b)*
50
250
Code-Switching
DA-X >
Eg-En En
Our work
50K
250
Jo-En + En
50
250
Ps-Fr + En
50
250
ANT Corpus
Chouigui al. (2021)
25.2K
3.1K
3.1K
CrossSum
Bhattacharjee et al. (2021)
37.3K
4.6K
4.7K
MSA -> MSA
Summarization
Mas
Varab and Schluter (2021)
4.6K
459
1.3K
XLSum
Hasan al. (2021)
37.5K
4.7K
4.7K
DA DA
MarSun
Gaanoun al. (2022)
16K
1.7K
1.9K
Arabic NTG
Nagoudi et al. (2022a)
93.3.5K
11.6K
11.6K
Title Generation
MSA MSA
XL.Sam
Hasan et al. (2021)
37.5K
4.7K
4.7K
ARCD
Mozannar al. (2019)1#
49.9K
693
702
MLQA
Lewis al. (2019y11
49.9K
517
5.3K
XQuAD
(2020y²
49.9K
5.08K
QA/QG
MSA
TyDiQA
Artetxe et (2020y
49.9K
5.08K
921
LAReQA
Roy al. (2020)
851
119
220
DAWQAS
Ismail and Nabhan Homsi (2018)
2.2K
318
645
EXAMS
Hardalov et ul. (2020)
7.9K
2.6K
13.5K
ANETAC
Ameur al. (2019)
75.9K
1K
3K
Transliteration
Arabizi
MSA
ATAR
Talatha (2021)
17.2K
2.1K
2.1K
NETTrans.
Merhav and (2018)
116K
14.5K
14.5K
Egy + MSA
3.8K
551
1.1K
Mag - MSA
3.1K
491
996
DA MSA
Mubarak (2018)
Text Rewriting
Lev - MSA
4.2K
594
1.2K
Gud > MSA
4.2K
594
1.2K
MSA + MSA
APGC
Alhafni et al. (2022)
40.4K
4.7K
11.3K
Diacritization
CA +
Fadel (2019)
50K
2.5K
2.5K
Datu2Text
Table + MSA
MD2T
Mille et al. (2020)
6K
900
680
MSA MSA
AEC
Naous al. (2020)
32.9K
1.8K
1.8K
Dialogue Generation
Egy Egy
2.1K
297
600
DA DA
Lev Lev
Naous (2023)
2.1K
297
Gud -
297
QALB 2014
Mohit et al. (2014)
19.4K
1K
968
GEC
MSA + MSA
QALB 2015
Rozovskaya et al. (2015)
310
154
158
ZAEBUC
Habash and Palfreyman (2022)
27K
3.3K
3.3K
ASEP
Cer (2017)
116.4K
6.1K
600
Paraphrase
MSA + MSA
Alian (2019)
808
TaPaCo
Scherrer (2020)
2.1K
299
605
Table B.2: Statistics of Dolphin benchmark across the different task clusters. For the QA task, the Arabic machine translated SQuAD (AR-XTREMFma. from Hu et al. (2020) as Train for ARCD, MIQQ,, and XQuAD. We also use AR-X IR-XTREME,, as Dev for XQuAD and TyiQA, respectively. For ASEP (Cer al., 2017) test set in the summarization task, we use AraParatuin and AraParape.-

---

ation. AraBART is designed for various NLP tasks, demonstrating robust performance across different tasks in the Arabic language.
mBART. A multilingual encoder-decoder model proposed by Liu et al. (2020). mBART is pre- trained by denoising full texts in 50 languages, in- cluding Arabic. Then, it is finetuned parallel MT data contains total of 230M parallel sentences under three settings: individually toward English and vice versa (i.e., many-to to-Enlishh, and English- to-many), or between multiple languages simulta- neously (many-to-manyy
m55 (Xue et al., 2020) is a multilingual variant of the of Text-to-Tex Transfer Transformer model (T5) (Raffel et al., 2019) that covers 101 languages. It is pretrained on a new Common Crawl-based dataset (~ 26.76TB), designed achieve SOTA performance on a variety of multilingual NLP tasks such as question answering, document summariza- tion, and MT.
mTO. (Muennighoff et al., 2022) is a group of sequence-to-sequuecee models ranging in size be- tween 300M to 13B parameters trained to investi- gate the cross-lingual generalization through mul- titask finetuning. The models are finetuned from preexisting mT5 (Xue et al., 2020) multilingual language models using cross-lingual task dataset called xP3. mTO models can execute human in- structions in many languages without any prior training.
D Leaderboard

---

DOLPHIN
Benchmark
Paper
Login
Overall Tasks
LeaderBoard
Cluster
Metric
Test Set
Model Name
Score
Show Results
Details
Code-Switching
DZ-FR
AmaARR
BG-EN to EN
mTD
7.19
+
MA-FR to FR
AMBART
16.14
PS-EN
YE-EN to EN
mTT
8.59
+
Discritization
CER
ADT
AraI5v2
1.36
Dialogue
Blcu
ABC
AmaART
1.71
Generation
DRG (EGY)
AmaAAR
0.35
DRG (GUL
mTO
1.02
DRG (LEV)
AraBART
0.57
Figure D.I: Dolphin main leaderboard allows showing detailed scores by all models given task.


Reviews:
[{"review_id": "rQwPaXhxOP", "reviewer": "Reviewer_rLjw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "> the paper introduces a new, comprehensive, and well-constructed benchmark with a clear comparison to existing benchmarks\n> the paper provides baselines, analysis, and evaluation framework that addresses and highlights strengths and weaknesses in existing works across different generative tasks which is very beneficial to researchers in this field\n\nThis is a high-impact paper well-suited to a high-impact conference. \n\n", "reasons_to_reject": "No reasons to reject.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1) The paper introduces a novel Arabic generative benchmark named Dolphin, which is a collection of 40 public datasets from four main categories: Modern Standard Arabic, Dialectical Arabic, Classical Arabic, and Arabizi. 2) Arabic and Multilingual models are evaluated on the benchmark to provide baselines across different tasks. 3) Well-defined tasks description, well-described methods, and good discussion  & analysis of results", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(2, 80) Abbreviation MT is first introduced without definition.\n(2, 96) Space between dolphin and design is missing\n(3) It is better to have Table 1 sorted by date"}, "extra_text": {}}, {"review_id": "MSSWDROayR", "reviewer": "Reviewer_5EDb", "summary": "", "questions": "1) Are there any major already available Arabic NLG datasets that you did not include? If so, why?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.\n\n**Reason to reject:** *\"The introduced leaderboard comes with an aggregate metric called Dolphin score, which is the macro-average of the scores across all tasks. While this is a way to do aggregate reporting, I wonder if it can conceal failures in the case of specific tasks. I understand this manner of having a single score is beneficial for leader boarding. But I'd also advocate for more granular performance reports. To be fair, this is something that's indeed supported by their leaderboard. I'm just mentioning it to emphasize it.\"*\n\n**Answer:** Thank you for the review. We appreciate your emphasis about the importance of including granular scores that allow revealing model performance on specific tasks. As you mention, our leaderboard does support such a level of granularity. That is, the leaderboard offers a score for each specific task (at the level of each individual dataset). This is in addition to the benchmark-level scores. As illustrated in Table 2 and Figure F.1 (Appendix F), our leaderboard introduces a suite of comprehensive options for the assessment of Arabic language models. These options encompass not only comparisons between models across individual tasks, but also gauge performance on task clusters focused on achieving lower scores (Dolphin_L Score), as well as task clusters where higher scores signify heightened performance (Dolphin_H Score). Since you clearly pointed out that the leaderboard has this level of granularity in your important statement reading “[t]o be fair, this is something that's indeed supported by their leaderboard. I'm just mentioning it to emphasize it”, we will appreciate you considering removing this as a reason to reject the paper. We very much appreciate you striving to be fair. \n\n**Q1:** *Are there any major already available Arabic NLG datasets that you did not include? If so, why?*\n\n**Answer:** To the best of our knowledge, there are no major publicly available datasets for Arabic NLG that we did not include. In fact, we have undertaken extensive searches to trace any publicly available dataset for inclusion in our benchmark. That is, our benchmark does not ignore any publicly available dataset we could locate even if such a dataset was not “major”. In other words, we strived to make Dolphin both exhaustive and inclusive. In addition, our intention is to enhance the benchmark and our interactive leaderboard in the future by adding datasets that will be newly created. \n\n**Typos:** We will correct the typos mentioned. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) I do believe this is a useful resource for Arabic NLG. I support efforts like this that provide standard and structured benchmarks for Languages other than English.\n2) The benchmark is designed in a way to honor easy public access, diversity, and referencing the original creators of a datasets if it's included from a prior work.\n3) This will help with the reproducibility of research in Arabic NLG.", "reasons_to_reject": "N/A", "questions_to_authors": "1) Are there any major already available Arabic NLG datasets that you did not include? If so, why?", "paper_topic_and_main_contributions": "The paper introduces a comprehensive benchmark for Arabic NLG comprising 40 datasets over 13 tasks. To create that they mostly rely on already publicly available datasets and they additionally create datasets for context switching and dialogue response generation. A leaderboard and an investigation of baseline performances on the benchmark are also included. The authors promote the benchmark as an important resource that can help boost research in Arabic NLG.\n\nThe introduced leaderboard comes with an aggregate metric called Dolphin score, which is the macro-average of the scores across all tasks. This manner of having a single score is beneficial for leaderboarding. Additionally, more granular performance reports are available.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) In the caption of Figure 1, I believe GEC should be replaced with Arabizi.\n2) Line 96 Dolphindesign -> Dolphin design\n3) Line 633 Dolphinmay -> Dolphin may"}, "extra_text": {}}, {"review_id": "9EHZoMEArg", "reviewer": "Reviewer_cCB7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.\n\n**Reason to reject #1:** *“similar to ARGEN”*\n\n**Answer:** In the context of comparing Dolphin with ARGEN, as highlighted within lines 119-123 and Table 1, several significant distinctions become apparent. Specifically, we observe the following:\n\n- Dolphin boasts a significantly larger dataset pool (~ 3X larger). Precisely, Dolphin comprises 40 datasets compared to only 13 datasets in ARGEN. Hence, Dolphin offers a total of 27 totally new datasets.\n- Dolphin's reach also extends to a wider array of task clusters, encompassing 13 clusters as opposed to ARGEN's seven clusters. Dolphin introduces six novel tasks: Arabic text diacritization, dialogue response generation, data-to-text conversion, grammatical error correction, text rewriting, and question answering. \n- Dolphin's datasets are drawn exclusively from publicly available sources, while ARGEN involves several non-public datasets such as Zbib et al. [2012] and Song et al. [2014]. As such, Dolphin avoids issues ARGEN suffers from such as challenges with (i) public distribution of the data and (ii) ease of evaluation. \n- Dolphin uniquely offers a benchmark leaderboard, a feature absent in ARGEN, providing real-time performance tracking and a dynamic evaluation environment. \n\n**Reason to reject #2:** *“the Dolphin benchmark just collected 40 datasets and combined them into a benchmark.”*\n\n**Answer:** Compilation of existing datasets to establish a benchmark is an important and established tradition in NLP that is shared, for example by other benchmarking efforts such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), GEMv1 (Gehrmann, 2021), GEMv2 (Gehrmann, 2022), MTG [Chen et al.,2022], etc. However, we would like to emphasize that our work involves much more than simple collection of existing datasets in that (1) we meticulously curate, select, standardize formatting and experimental conditions of a large number of datasets; (2) we thematically categorize the datasets across several task clusters, (3) we characterize the extent to which the datasets represent a diverse array of challenging NLG tasks across different Arabic varieties (CA, dialects, and MSA); (4) We create six new human-written (natural) code-switched parallel test datasets that we incorporate into the benchmark; (5) we exploit the datasets to compare different Arabic language models suited for NLG; and (6) we create a highly modular and interactive leaderboard to allow for measurement of progress on Arabic and multilingual NLG. \n\n**Reason to reject #3:** *“ The design principles used in the papers are also mentioned in the ARGEN benchmark paper.”*\n\n**Answer:** The ARGEN benchmark paper does not discuss any design principles. Again, our work is significantly different from the work introducing the ARGEN benchmark and Dolphin itself is completely different from ARGEN as we emphasize in our response to the first point above. For example, ARGEN does not entertain dataset accessibility (it has private datasets) nor ease of meaningful comparison via a public leaderboard. In addition, it does not claim representativeness of Arabic varieties (e.g., it has no coverage of CA).  \n\n**Reason to reject #4:** *“The discussion of model evaluation on Dolphin does not consider the signatures of the Dolphin benchmark, including diverse and inclusive, or any specified tasks. The \"Model Computational Costs\" part of the Discussion is meaningless. “*\n\n**Answer:** Our discussion of Dolphin does cover it being diverse and inclusive (Figure 1, lines 176-194, and Table C.1). However, we will strengthen and deepen this discussion. We will also offer a stronger and explicit comparison between performance of each of the models on the different varieties (CA, DA, MSA). \n\nWe provide the section on model computational cost since we believe it is useful for practitioners who have limited compute resources. It can be helpful for choosing which model to use under which compute budget. We will explain this motivation along with a thorough analysis and comparison of the computational expenses of the different models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's organization and presentation are commendable overall.\n\nThe work exhibits many experiment results of multilingual pre-trained language models on their novel benchmark.", "reasons_to_reject": "Similar to ARGEN, the Dolphin benchmark just collected 40 datasets and combined them into a benchmark. The design principles used in the papers are also mentioned in the ARGEN benchmark paper.\n\nThe discussion of model evaluation on Dolphin does not consider the signatures of the Dolphin benchmark, including diverse and inclusive, or any specified tasks.\n\nThe \"Model Computational Costs\" part of the Discussion is meaningless.\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper introduced a benchmark for Arabic natural language generation (NLG) consisting of 13 tasks curated from 40 datasets. The authors evaluate their benchmark on multiple multilingual pre-trained language models for both finetuning and k-shot settings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a comprehensive benchmark for Arabic NLG comprising 40 datasets over 13 tasks. To create that they mostly rely on already publicly available datasets and they additionally create datasets for context switching and dialogue response generation. A leaderboard and an investigation of baseline performances on the benchmark are also included. The authors promote the benchmark as an important resource that can help boost research in Arabic NLG.

Reasons To Accept:
- Introduction of a new, comprehensive, and well-constructed benchmark with a clear comparison to existing benchmarks 
- The paper provides baselines, analysis, and evaluation framework that addresses and highlights strengths and weaknesses in existing works across different generative tasks which is very beneficial to researchers in this field
- This is a high-impact paper well-suited to a high-impact conference.
-Useful resource for Arabic NLG. 
- Public access and diversity
- Help with the reproducibility of research in Arabic NLG.

Reasons To Reject:
- Similar to ARGEN. The design principles used in the papers are also mentioned in the ARGEN benchmark paper.

In my opinion the reason to reject is not fair, and the answer written by the authors is explicit:
"In the context of comparing Dolphin with ARGEN, as highlighted within lines 119-123 and Table 1, several significant distinctions become apparent. Specifically, we observe the following:
- Dolphin boasts a significantly larger dataset pool (~ 3X larger). Precisely, Dolphin comprises 40 datasets compared to only 13 datasets in ARGEN. Hence, Dolphin offers a total of 27 totally new datasets.
- Dolphin's reach also extends to a wider array of task clusters, encompassing 13 clusters as opposed to ARGEN's seven clusters. Dolphin introduces six novel tasks: Arabic text diacritization, dialogue response generation, data-to-text conversion, grammatical error correction, text rewriting, and question answering.
- Dolphin's datasets are drawn exclusively from publicly available sources, while ARGEN involves several non-public datasets such as Zbib et al. [2012] and Song et al. [2014]. As such, Dolphin avoids issues ARGEN suffers from such as challenges with (i) public distribution of the data and (ii) ease of evaluation.
- Dolphin uniquely offers a benchmark leaderboard, a feature absent in ARGEN, providing real-time performance tracking and a dynamic evaluation environment."
Decision: Accept-Findings