IVUL-KAUST/MOLE huggingface dataset:
Dataset: FLORES-101
Description: The FLORES-101 evaluation benchmark consists of 3001 sentences extracted from English Wikipedia and covers various topics and domains. These sentences have been translated into 101 languages by professional translators through a carefully controlled process.
Language: multilingual
Dialect: Modern Standard Arabic
Script: Arab
Domain: wikipedia, books, news articles
Form: text
Size: 3001 sentences
Provider: Facebook
Tasks: machine translation
License: CC BY-SA 4.0
Year: 2021
Authors: Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Guillaume Wenzek, Da Ju, Sanjan Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, Angela Fan
Affiliations: Facebook AI Research
Abstract: One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.
Dataset: Transliteration
Description: Arabic-English transliteration dataset mined from Wikipedia.
Language: multilingual
Dialect: Modern Standard Arabic
Script: Arab-Latin
Domain: wikipedia
Form: text
Size: 15898 tokens
Provider: Google
Tasks: transliteration, machine translation
License: Apache-2.0
Year: 2016
Authors: Mihaela Rosca, Thomas Breuel
Affiliations: Google
Abstract: Transliteration is a key component of machine translation systems and software internationalization. This paper demonstrates that neural sequence-to-sequence models obtain state of the art or close to state of the art results on existing datasets. In an effort to make machine transliteration accessible, we open source a new Arabic to English transliteration dataset and our trained models.
Dataset: ADI-5
Description: This will be divided across the five major Arabic dialects; Egyptian (EGY), Levantine (LAV), Gulf (GLF), North African (NOR), and Modern Standard Arabic (MSA)
Language: ar
Dialect: mixed
Script: Arab
Domain: TV Channels
Form: audio
Size: 74.5 hours
Subsets: Name: Egyptian
Dialect: Egypt
Volume: 14.4
Unit: hours, Name: Gulf
Dialect: Gulf
Volume: 14.1
Unit: hours, Name: Levantine
Dialect: Levant
Volume: 14.3
Unit: hours, Name: MSA
Dialect: Modern Standard Arabic
Volume: 14.3
Unit: hours, Name: North African
Dialect: North Africa
Volume: 14.6
Unit: hours
Provider: QCRI
Tasks: dialect identification
License: MIT License
Year: 2016
Authors: A. Ali, Najim Dehak, P. Cardinal, Sameer Khurana, S. Yella, James R. Glass, P. Bell, S. Renals
Abstract: We investigate different approaches for dialect identification in Arabic broadcast speech, using phonetic, lexical features obtained from a speech recognition system, and acoustic features using the i-vector framework. We studied both generative and discriminate classifiers, and we combined these features using a multi-class Support Vector Machine (SVM). We validated our results on an Arabic/English language identification task, with an accuracy of 100%. We used these features in a binary classifier to discriminate between Modern Standard Arabic (MSA) and Dialectal Arabic, with an accuracy of 100%. We further report results using the proposed method to discriminate between the five most widely used dialects of Arabic: namely Egyptian, Gulf, Levantine, North African, and MSA, with an accuracy of 52%. We discuss dialect identification errors in the context of dialect code-switching between Dialectal Arabic and MSA, and compare the error pattern between manually labeled data, and the output from our classifier. We also release the train and test data as standard corpus for dialect identification.
Dataset: Maknuune
Description: A large open lexicon for the Palestinian Arabic dialect. Maknuune has over 36K entries from 17K lemmas,and 3.7K roots. All entries include diacritized Arabic orthography, phonological transcription and English glosses.
Language: multilingual
Dialect: Palestine
Script: Arab-Latin
Domain: captions, public datasets, other
Form: text
Size: 36302 tokens
Provider: New York University Abu Dhabi, University of Oxford, UNRWA
Tasks: morphological analysis, lexicon analysis
License: CC BY-SA 4.0
Year: 2022
Authors: Shahd Dibas, Christian Khairallah, Nizar Habash, Omar Fayez Sadi, Tariq Sairafy, Karmel Sarabta, Abrar Ardah
Affiliations: NYUAD, University of Oxford, UNRWA
Abstract: We present Maknuune, a large open lexicon for the Palestinian Arabic dialect. Maknuune has over 36K entries from 17K lemmas, and 3.7K roots. All entries include diacritized Arabic orthography, phonological transcription and English glosses. Some entries are enriched with additional information such as broken plurals and templatic feminine forms, associated phrases and collocations, Standard Arabic glosses, and examples or notes on grammar, usage, or location of collected entry.
Dataset: EmojisAnchors
Description: Fine-Grained Hate Speech Detection on Arabic Twitter
Language: ar
Dialect: mixed
Script: Arab
Domain: social media, public datasets
Form: text
Size: 12698 sentences
Provider: QCRI, University of Pittsburgh
Tasks: offensive language detection
License: custom
Year: 2022
Authors: Hamdy Mubarak, Hend Al-Khalifa, AbdulMohsen Al-Thubaity
Affiliations: Qatar Computing Research Institute, King Saud University, King Abdulaziz City for Science and Technology (KACST)
Abstract: We introduce a generic, language-independent method to collect a large percentage of offensive and hate tweets regardless of their topics or genres. We harness the extralinguistic information embedded in the emojis to collect a large number of offensive tweets. We apply the proposed method on Arabic tweets and compare it with English tweets - analysing key cultural differences. We observed a constant usage of these emojis to represent offensiveness throughout different timespans on Twitter. We manually annotate and publicly release the largest Arabic dataset for offensive, fine-grained hate speech, vulgar and violence content. Furthermore, we benchmark the dataset for detecting offensiveness and hate speech using different transformer architectures and perform in-depth linguistic analysis. We evaluate our models on external datasets - a Twitter dataset collected using a completely different method, and a multi-platform dataset containing comments from Twitter, YouTube and Facebook, for assessing generalization capability. Competitive results on these datasets suggest that the data collected using our method captures universal characteristics of offensive language. Our findings also highlight the common words used in offensive communications, common targets for hate speech, specific patterns in violence tweets; and pinpoint common classification errors that can be attributed to limitations of NLP models. We observe that even state-of-the-art transformer models may fail to take into account culture, background and context or understand nuances present in real-world data such as sarcasm.
Dataset: Calliar
Description: Calliar is a dataset for Arabic calligraphy. The dataset consists of 2500 json files that contain strokes manually annotated for Arabic calligraphy. This repository contains the dataset for the following paper
Language: ar
Dialect: Modern Standard Arabic
Script: Arab
Domain: web pages
Form: images
Size: 2500 images
Provider: ARBML
Tasks: optical character recognition
License: MIT License
Year: 2021
Authors: Zaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb & Yousif Ahmed Al-Wajih
Affiliations: KFUPM, KFUPM, KFUPM, KFUPM
Abstract: Calligraphy is an essential part of the Arabic heritage and culture. It has been used in the past for the decoration of houses and mosques. Usually, such calligraphy is designed manually by experts with aesthetic insights. In the past few years, there has been a considerable effort to digitize such type of art by either taking a photo of decorated buildings or drawing them using digital devices. The latter is considered an online form where the drawing is tracked by recording the apparatus movement, an electronic pen for instance, on a screen. In the literature, there are many offline datasets collected with a diversity of Arabic styles for calligraphy. However, there is no available online dataset for Arabic calligraphy. In this paper, we illustrate our approach for the collection and annotation of an online dataset for Arabic calligraphy called Calliar that consists of 2,500 sentences. Calliar is annotated for stroke, character, word and sentence level prediction.
Dataset: LABR
Description: A large Arabic book review dataset for sentiment analysis
Language: ar
Dialect: mixed
Script: Arab
Domain: social media, reviews
Form: text
Size: 63257 sentences
Provider: Cairo University
Tasks: review classification, sentiment analysis
License: GPL-2.0
Year: 2015
Authors: Mahmoud Nabil, Mohamed Aly, Amir F. Atiya
Affiliations: Cairo University, Cairo University, Cairo University
Abstract: We introduce LABR, the largest sentiment analysis dataset to-date for the Arabic language. It consists of over 63,000 book reviews, each rated on a scale of 1 to 5 stars. We investigate the properties of the dataset, and present its statistics. We explore using the dataset for two tasks: (1) sentiment polarity classification; and (2) ratings classification. Moreover, we provide standard splits of the dataset into training, validation and testing, for both polarity and ratings classification, in both balanced and unbalanced settings. We extend our previous work by performing a comprehensive analysis on the dataset. In particular, we perform an extended survey of the different classifiers typically used for the sentiment polarity classification problem. We also construct a sentiment lexicon from the dataset that contains both single and compound sentiment words and we explore its effectiveness. We make the dataset and experimental details publicly available.
Dataset: ACVA
Description: ACVA is a Yes-No question dataset, comprising over 8000 questions, generated by GPT-3.5 Turbo from 50 designed Arabic topics to assess model alignment with Arabic values and cultures
Language: ar
Dialect: Modern Standard Arabic
Script: Arab
Domain: LLM
Form: text
Size: 8000 sentences
Provider: FreedomIntelligence
Tasks: question answering
License: Apache-2.0
Year: 2023
Authors: Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu
Abstract: This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.   Comprehensive evaluations reveal that the resulting model, dubbed `AceGPT', sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in https://github.com/FreedomIntelligence/AceGPT.
Dataset: ATHAR
Description: The ATHAR dataset comprises 66,000 translation pairs from Classical Arabic to English. It spans a wide array of subjects, aiming to enhance the development of NLP models specialized in Classical Arabic.
Language: multilingual
Dialect: Classical Arabic
Script: Arab
Domain: books
Form: text
Size: 66000 sentences
Provider: ADAPT/DCU
Tasks: machine translation
License: CC BY-SA 4.0
Year: 2024
Authors: Mohammed Khalil, Mohammed Sabry
Affiliations: Independent Researcher, ADAPT/DCU
Abstract: Classical Arabic represents a significant era, encompassing the golden age of Arab culture, philosophy, and scientific literature. With a broad consensus on the importance of translating these literatures to enrich knowledge dissemination across communities, the advent of large language models (LLMs) and translation systems offers promising tools to facilitate this goal. However, we have identified a scarcity of translation datasets in Classical Arabic, which are often limited in scope and topics, hindering the development of high-quality translation systems. In response, we present the ATHAR dataset, comprising 66,000 high-quality Classical Arabic to English translation samples that cover a wide array of subjects including science, culture, and philosophy. Furthermore, we assess the performance of current state-of-the-art LLMs under various settings, concluding that there is a need for such datasets in current systems. Our findings highlight how models can benefit from fine-tuning or incorporating this dataset into their pretraining pipelines. The dataset is publicly available on the HuggingFace Data Hub at \url{https://huggingface.co/datasets/mohamed-khalil/ATHAR}.
Dataset: OpenITI-proc
Description: A linguistically annotated version of the OpenITI corpus, with annotations for lemmas, POS tags, parse trees, and morphological segmentation
Language: ar
Dialect: Classical Arabic
Script: Arab
Domain: public datasets, books
Form: text
Size: 7144 documents
Tasks: text generation, language modeling
License: CC BY 4.0
Year: 2018
Authors: Yonatan Belinkov, Alexander Magidow, Alberto Barrón-Cedeño, Avi Shmidman, Maxim Romanov
Abstract: Arabic is a widely-spoken language with a long and rich history, but existing corpora and language technology focus mostly on modern Arabic and its varieties. Therefore, studying the history of the language has so far been mostly limited to manual analyses on a small scale. In this work, we present a large-scale historical corpus of the written Arabic language, spanning 1400 years. We describe our efforts to clean and process this corpus using Arabic NLP tools, including the identification of reused text. We study the history of the Arabic language using a novel automatic periodization algorithm, as well as other techniques. Our findings confirm the established division of written Arabic into Modern Standard and Classical Arabic, and confirm other established periodizations, while suggesting that written Arabic may be divisible into still further periods of development.
Dataset: AraDangspeech
Description: Dangerous speech detection
Language: ar
Dialect: mixed
Script: Arab
Domain: social media
Form: text
Size: 5011 sentences
Provider: The University of British Columbia
Tasks: offensive language detection
License: unknown
Year: 2020
Authors: Ali Alshehri, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed
Affiliations: The University of British Columbia
Abstract: Social media communication has become a significant part of daily activity in modern societies. For this reason, ensuring safety in social media platforms is a necessity. Use of dangerous language such as physical threats in online environments is a somewhat rare, yet remains highly important. Although several works have been performed on the related issue of detecting offensive and hateful language, dangerous speech has not previously been treated in any significant way. Motivated by these observations, we report our efforts to build a labeled dataset for dangerous speech. We also exploit our dataset to develop highly effective models to detect dangerous content. Our best model performs at 59.60% macro F1, significantly outperforming a competitive baseline.
Dataset: Arabic-Hebrew TED Talks Parallel Corpus
Description: This dataset consists of 2023 TED talks with aligned Arabic and Hebrew subtitles. Sentences were rebuilt and aligned using English as a pivot to improve accuracy, offering a valuable resource for Arabic-Hebrew machine translation tasks.
Language: multilingual
Dialect: Modern Standard Arabic
Script: Arab
Domain: captions, public datasets
Form: text
Size: 225000 sentences
Provider: FBK
Tasks: machine translation
License: unknown
Year: 2016
Authors: Mauro Cettolo
Affiliations: Fondazione Bruno Kessler (FBK)
Abstract: We describe an Arabic-Hebrew parallel corpus of TED talks built upon WIT3, the Web inventory that repurposes the original content of the TED website in a way which is more convenient for MT researchers. The benchmark consists of about 2,000 talks, whose subtitles in Arabic and Hebrew have been accurately aligned and rearranged in sentences, for a total of about 3.5M tokens per language. Talks have been partitioned in train, development and test sets similarly in all respects to the MT tasks of the IWSLT 2016 evaluation campaign. In addition to describing the benchmark, we list the problems encountered in preparing it and the novel methods designed to solve them. Baseline MT results and some measures on sentence length are provided as an extrinsic evaluation of the quality of the benchmark.
Dataset: ARASPIDER
Description: AraSpider is a translated version of the Spider dataset, which is commonly used for semantic parsing and text-to-SQL generation. The dataset includes 200 databases across 138 domains with 10,181 questions and 5,693 unique complex SQL queries.
Language: ar
Dialect: Modern Standard Arabic
Script: Arab
Domain: public datasets, LLM
Form: text
Size: 10181 sentences
Provider: Egypt-Japan University of Science and Technology
Tasks: semantic parsing, text to SQL
License: MIT License
Year: 2024
Authors: Ahmed Heakl, Youssef Mohamed, Ahmed B. Zaky
Affiliations: Egypt-Japan University of Science and Technology, Egypt-Japan University of Science and Technology, Egypt-Japan University of Science and Technology
Abstract: This study presents AraSpider, the first Arabic version of the Spider dataset, aimed at improving natural language processing (NLP) in the Arabic-speaking community. Four multilingual translation models were tested for their effectiveness in translating English to Arabic. Additionally, two models were assessed for their ability to generate SQL queries from Arabic text. The results showed that using back translation significantly improved the performance of both ChatGPT 3.5 and SQLCoder models, which are considered top performers on the Spider dataset. Notably, ChatGPT 3.5 demonstrated high-quality translation, while SQLCoder excelled in text-to-SQL tasks. The study underscores the importance of incorporating contextual schema and employing back translation strategies to enhance model performance in Arabic NLP tasks. Moreover, the provision of detailed methodologies for reproducibility and translation of the dataset into other languages highlights the research's commitment to promoting transparency and collaborative knowledge sharing in the field. Overall, these contributions advance NLP research, empower Arabic-speaking researchers, and enrich the global discourse on language comprehension and database interrogation.
Dataset: HellaSwag
Description: HellaSwag is a dataset for physically situated commonsense reasoning.
Language: en
Domain: captions, public datasets, wikipedia
Form: text
Size: 70000 sentences
Provider: Allen Institute of Artificial Intelligence
Tasks: natural language inference
License: MIT License
Year: 2019
Authors: Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi
Affiliations: University of Washington, Allen Institute of Artificial Intelligence
Abstract: Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference?   In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.   Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Dataset: GPQA
Description: A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. The questions are designed to be difficult for both state-of-the-art AI systems and skilled non-experts, even with access to the web.
Language: en
Domain: other
Form: text
Size: 448 sentences
Provider: New York University, Cohere, Anthropic
Tasks: multiple choice question answering
License: CC BY 4.0
Year: 2023
Authors: David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
Affiliations: New York University, Cohere, Anthropic, PBC
Abstract: We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.