IVUL-KAUST/MOLE huggingface dataset:
Dataset: SEAHORSE
Description: SEAHORSE is a large-scale dataset for multilingual, multifaceted summarization evaluation. It consists of 96,645 summaries with human ratings along 6 quality dimensions: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness. It covers 6 languages, 9 systems, and 4 summarization datasets.
Language: German, English, Spanish, Russian, Turkish, Vietnamese
Domain: public datasets
Form: text
Size: 96645 sentences
Subsets: Name: de
Language: German
Volume: 14591
Unit: sentences, Name: en
Language: English
Volume: 22339
Unit: sentences, Name: es
Language: Spanish
Volume: 14749
Unit: sentences, Name: ru
Language: Russian
Volume: 14542
Unit: sentences, Name: tr
Language: Turkish
Volume: 15418
Unit: sentences, Name: vi
Language: Vietnamese
Volume: 15006
Unit: sentences
Provider: Google
Tasks: summarization
License: CC BY 4.0
Year: 2023
Authors: Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur P. Parikh
Affiliations: Google DeepMind, Google Research
Abstract: Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems and 4 datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2022). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.
Dataset: Mintaka
Description: Mintaka is a large, complex, naturally-elicited, and multilingual question answering dataset. It contains 20,000 English question-answer pairs, which have been translated into 8 other languages, totaling 180,000 samples. The dataset is annotated with Wikidata entities and includes 8 types of complex questions.
Language: English, Arabic, French, German, Hindi, Italian, Japanese, Portuguese, Spanish
Domain: wikipedia
Form: text
Size: 180000 sentences
Subsets: Name: English
Language: English
Volume: 20000
Unit: sentences, Name: Arabic
Language: Arabic
Volume: 20000
Unit: sentences, Name: French
Language: French
Volume: 20000
Unit: sentences, Name: German
Volume: 20000
Language: German
Unit: sentences, Name: Hindi
Volume: 20000
Language: Hindi
Unit: sentences, Name: Italian
Volume: 20000
Language: Italian
Unit: sentences, Name: Japanese
Language: Japanese
Volume: 20000
Unit: sentences, Name: Portuguese
Language: Portuguese
Volume: 20000
Unit: sentences, Name: Spanish
Language: Spanish
Volume: 20000
Unit: sentences
Provider: Amazon
Tasks: question answering
License: CC BY 4.0
Year: 2022
Authors: Priyanka Sen, Alham Fikri Aji, Amir Saffari
Affiliations: Amazon Alexa AI
Abstract: We introduce Mintaka, a complex, natural, and multilingual dataset designed for experimenting with end-to-end question-answering models. Mintaka is composed of 20,000 question-answer pairs collected in English, annotated with Wikidata entities, and translated into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka includes 8 types of complex questions, including superlative, intersection, and multi-hop questions, which were naturally elicited from crowd workers. We run baselines over Mintaka, the best of which achieves 38% hits@1 in English and 31% hits@1 multilingually, showing that existing models have room for improvement. We release Mintaka at https://github.com/amazon-research/mintaka.
Dataset: Multi2WOZ
Description: A multilingual, multi-domain task-oriented dialog (TOD) dataset in Arabic, Chinese, German, and Russian. It was created by translating and manually post-editing the 2,000 development and test dialogs from the English MultiWOZ 2.1 dataset, enabling reliable cross-lingual transfer evaluation.
Language: Arabic, Chinese, German, Russian
Domain: public datasets
Form: text
Size: 118000 sentences
Subsets: Name: Arabic
Language: Arabic
Volume: 29500
Unit: sentences, Name: Chinese
Language: Chinese
Volume: 29500
Unit: sentences, Name: German
Language: German
Volume: 29500
Unit: sentences, Name: Russian
Language: Russian
Volume: 29500
Unit: sentences
Provider: University of Mannheim
Tasks: instruction tuning
License: MIT License
Year: 2022
Authors: Chia-Chien Hung, Anne Lauscher, Ivan Vulic´, Simone Paolo Ponzetto, Goran Glavasˇ
Affiliations: Data and Web Science Group, University of Mannheim, Germany, MilaNLP, Bocconi University, Italy, LTL, University of Cambridge, UK, CAIDAS, University of Wu¨rzburg, Germany
Abstract: Research on (multi-domain) task-oriented dialog (TOD) has predominantly focused on the English language, primarily due to the shortage of robust TOD datasets in other languages, preventing the systematic investigation of cross-lingual transfer for this crucial NLP application area. In this work, we introduce Multi2WOZ, a new multilingual multi-domain TOD dataset, derived from the well-established English dataset MultiWOZ, that spans four typologically diverse languages: Chinese, German, Arabic, and Russian. In contrast to concurrent efforts, Multi2WOZ contains gold-standard dialogs in target languages that are directly comparable with development and test portions of the English dataset, enabling reliable and comparative estimates of cross-lingual transfer performance for TOD. We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks. Using such conversational PrLMs specialized for concrete target languages, we systematically benchmark a number of zero-shot and few-shot cross-lingual transfer approaches on two standard TOD tasks: Dialog State Tracking and Response Retrieval. Our experiments show that, in most setups, the best performance entails the combination of (I) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task. Most importantly, we show that our conversational specialization in the target language allows for an exceptionally sample-efficient few-shot transfer for downstream TOD tasks.
Dataset: MTOP
Description: MTOP is a multilingual, almost-parallel dataset for task-oriented semantic parsing. It comprises 100k annotated utterances in 6 languages (English, German, French, Spanish, Hindi, Thai) across 11 domains. The dataset is designed to handle complex, nested queries through a compositional representation scheme.
Language: English, German, French, Spanish, Hindi, Thai
Domain: other
Form: text
Size: 104445 sentences
Subsets: Name: English
Volume: 22288
Unit: sentences
Language: English, Name: German
Volume: 18788
Unit: sentences
Language: German, Name: French
Volume: 16584
Unit: sentences
Language: French, Name: Spanish
Volume: 15459
Unit: sentences
Language: Spanish, Name: Hindi
Volume: 16131
Unit: sentences
Language: Hindi, Name: Thai
Volume: 15195
Unit: sentences
Language: Thai
Provider: Facebook
Tasks: named entity recognition, intent classification
License: unknown
Year: 2021
Authors: Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, Yashar Mehdad
Affiliations: Facebook
Abstract: Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.
Dataset: X-RiSAWOZ
Description: A multi-domain, large-scale, and high-quality task-oriented dialogue benchmark, produced by translating the Chinese RiSAWOZ data to four diverse languages: English, French, Hindi, and Korean; and one code-mixed English-Hindi language. It is an end-to-end dataset for building fully-functioning agents.
Language: Chinese, English, French, Hindi, Korean
Domain: public datasets
Form: text
Size: 90000 sentences
Subsets: Name: Chinese
Language: Chinese
Volume: 18000
Unit: sentences, Name: English
Language: English
Volume: 18000
Unit: sentences, Name: French
Language: French
Volume: 18000
Unit: sentences, Name: Hindi
Language: Hindi
Volume: 18000
Unit: sentences, Name: Korean
Language: Korean
Volume: 18000
Unit: sentences
Provider: Stanford University
Tasks: instruction tuning
License: custom
Year: 2023
Authors: Mehrad Moradshahi, Tianhao Shen, Kalika Bali, Monojit Choudhury, Gaël de Chalendar, Anmol Goel, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Nasredine Semmar, Sina J. Semnani, Jiwon Seo, Vivek Seshadri, Manish Shrivastava, Michael Sun, Aditya Yadavalli, Chaobin You, Deyi Xiong, Monica S. Lam
Affiliations: Stanford University, Tianjin University, Microsoft, Université Paris-Saclay, International Institute of Information Technology, Hyderabad, Hanyang University, Karya Inc.
Abstract: Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language. To reduce the cost, we apply manual editing to automatically translated data. We create a new multilingual benchmark, X-RiSAWOZ, by translating the Chinese RiSAWOZ to 4 languages: English, French, Hindi, Korean; and a code-mixed English-Hindi language. X-RiSAWOZ has more than 18,000 human-verified dialogue utterances for each language, and unlike most multilingual prior work, is an end-to-end dataset for building fully-functioning agents.   The many difficulties we encountered in creating X-RiSAWOZ led us to develop a toolset to accelerate the post-editing of a new language dataset after translation. This toolset improves machine translation with a hybrid entity alignment technique that combines neural with dictionary-based methods, along with many automated and semi-automated validation checks.   We establish strong baselines for X-RiSAWOZ by training dialogue agents in the zero- and few-shot settings where limited gold data is available in the target language. Our results suggest that our translation and post-editing methodology and toolset can be used to create new high-quality multilingual dialogue agents cost-effectively. Our dataset, code, and toolkit are released open-source.
Dataset: PRESTO
Description: PRESTO is a public, multilingual dataset of over 550K contextual conversations between humans and virtual assistants for parsing realistic task-oriented dialogs. It contains challenges like disfluencies, code-switching, and user revisions, and provides structured context (contacts, lists) for each example across six languages.
Language: German, English, Spanish, French, Hindi, Japanese
Domain: other
Form: text
Size: 552924 sentences
Subsets: Language: German
Name: German
Volume: 83584
Unit: sentences, Name: English
Unit: sentences
Language: English
Volume: 95671, Unit: sentences
Name: Spanish
Language: Spanish
Volume: 96164, Volume: 95870
Unit: sentences
Language: French
Name: French, Unit: sentences
Name: Hindi
Volume: 72107
Language: Hindi, Name: Japanese
Volume: 109528
Unit: sentences
Language: Japanese
Provider: Google Inc.
Tasks: intent classification, instruction tuning
License: CC BY 4.0
Year: 2023
Authors: Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Kyle He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu
Affiliations: Google Inc., University of Rochester, University of California, Santa Barbara, Columbia University
Abstract: Research interest in task-oriented dialogs has increased as systems such as Google Assistant, Alexa and Siri have become ubiquitous in everyday life. However, the impact of academic research in this area has been limited by the lack of datasets that realistically capture the wide array of user pain points. To enable research on some of the more challenging aspects of parsing realistic conversations, we introduce PRESTO, a public dataset of over 550K contextual multilingual conversations between humans and virtual assistants. PRESTO contains a diverse array of challenges that occur in real-world NLU tasks such as disfluencies, code-switching, and revisions. It is the only large scale human generated conversational parsing dataset that provides structured context such as a user's contacts and lists for each example. Our mT5 model based baselines demonstrate that the conversational phenomenon present in PRESTO are challenging to model, which is further pronounced in a low-resource setup.
Dataset: LAHM
Description: A large-scale, semi-supervised dataset for multilingual and multi-domain hate speech identification. It contains nearly 300k tweets across 6 languages (English, Hindi, Arabic, French, German, Spanish) and 5 domains (Abuse, Racism, Sexism, Religious Hate, Extremism), created using a 3-layer annotation pipeline.
Language: English, Hindi, Arabic, French, German, Spanish
Domain: social media, news articles, public datasets
Form: text
Size: 227836 sentences
Subsets: Name: English
Volume: 105120
Unit: sentences
Language: English, Name: Hindi
Volume: 32734
Unit: sentences
Language: Hindi, Name: Arabic
Volume: 5394
Unit: sentences
Language: Arabic, Name: French
Volume: 20809
Unit: sentences
Language: French, Name: German
Volume: 8631
Unit: sentences
Language: German, Name: Spanish
Volume: 55148
Unit: sentences
Language: Spanish
Provider: Logically.ai
Tasks: offensive language detection
License: unknown
Year: 2023
Authors: Ankit Yadav, Shubham Chandel, Sushant Chatufale, Anil Bandhakavi
Affiliations: Logically.ai
Abstract: Current research on hate speech analysis is typically oriented towards monolingual and single classification tasks. In this paper, we present a new multilingual hate speech analysis dataset for English, Hindi, Arabic, French, German and Spanish languages for multiple domains across hate speech - Abuse, Racism, Sexism, Religious Hate and Extremism. To the best of our knowledge, this paper is the first to address the problem of identifying various types of hate speech in these five wide domains in these six languages. In this work, we describe how we created the dataset, created annotations at high level and low level for different domains and how we use it to test the current state-of-the-art multilingual and multitask learning approaches. We evaluate our dataset in various monolingual, cross-lingual and machine translation classification settings and compare it against open source English datasets that we aggregated and merged for this task. Then we discuss how this approach can be used to create large scale hate-speech datasets and how to leverage our annotations in order to improve hate speech detection and classification in general.
Dataset: MARC
Description: Amazon product reviews dataset for multilingual text classification. The dataset contains reviews in English, Japanese, German, French, Chinese and Spanish, collected between November 1, 2015 and November 1, 2019
Language: Japanese, English, German, French, Spanish, Chinese
Domain: reviews
Form: text
Size: 12600000 sentences
Subsets: Name: English
Volume: 2100000
Unit: sentences
Language: English, Name: Japanese
Volume: 2100000
Unit: sentences
Language: Japanese, Name: German
Volume: 2100000
Unit: sentences
Language: German, Name: French
Volume: 2100000
Unit: sentences
Language: French, Name: Spanish
Volume: 2100000
Unit: sentences
Language: Spanish, Name: Chinese
Volume: 2100000
Unit: sentences
Language: Chinese
Provider: Amazon
Tasks: sentiment analysis, review classification
License: custom
Year: 2020
Authors: Phillip Keung, Yichao Lu, Gyorgy Szarvas, Noah A. Smith
Affiliations: Amazon, Washington University
Abstract: We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification. The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019. Each record in the dataset contains the review text, the review title, the star rating, an anonymized reviewer ID, an anonymized product ID, and the coarse-grained product category (e.g., 'books', 'appliances', etc.) The corpus is balanced across the 5 possible star ratings, so each rating constitutes 20% of the reviews in each language. For each language, there are 200,000, 5,000, and 5,000 reviews in the training, development, and test sets, respectively. We report baseline results for supervised text classification and zero-shot cross-lingual transfer learning by fine-tuning a multilingual BERT model on reviews data. We propose the use of mean absolute error (MAE) instead of classification accuracy for this task, since MAE accounts for the ordinal nature of the ratings.
Dataset: MLSUM
Description: MLSUM is a large-scale multilingual summarization dataset with over 1.5 million article/summary pairs in French, German, Spanish, Russian, and Turkish. Collected from online newspapers, it is designed to complement the English CNN/Daily Mail dataset, enabling new research in cross-lingual summarization.
Language: French, German, Spanish, Russian, Turkish
Domain: news articles, web pages
Form: text
Size: 1259070 documents
Subsets: Name: FR
Volume: 424763
Unit: documents
Language: French, Name: DE
Volume: 242982
Unit: documents
Language: German, Name: ES
Volume: 290645
Unit: documents
Language: Spanish, Name: RU
Volume: 27063
Unit: documents
Language: Russian, Name: TR
Volume: 273617
Unit: documents
Language: Turkish
Provider: reciTAL, Sorbonne Université, CNRS
Tasks: summarization
License: custom
Year: 2020
Authors: Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano
Affiliations: reciTAL, Paris, France, Sorbonne Université, CNRS, LIP6, F-75005 Paris, France, CNRS, France
Abstract: We present MLSUM, the first large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian, Turkish. Together with English newspapers from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. We report cross-lingual comparative analyses based on state-of-the-art systems. These highlight existing biases which motivate the use of a multi-lingual dataset.