IVUL-KAUST/MOLE huggingface dataset:
Dataset: RusCode
Description: RusCode is a benchmark dataset designed to evaluate the quality of text-to-image generation models on their understanding of the Russian cultural code. It consists of 1250 text prompts in Russian, with English translations, covering 19 categories that represent Russian visual culture, including art, folklore, famous personalities, and scientific achievements.
Language: multilingual
Domain: other
Form: text
Size: 1250 sentences
Provider: SberAI
Tasks: instruction tuning
License: MIT License
Year: 2025
Authors: Viacheslav Vasilev, Julia Agafonova, Nikolai Gerasimenko, Alexander Kapitanov, Polina Mikhailova, Evelina Mironova, Denis Dimitrov
Affiliations: SberAI, MIPT, ITMO University, SberDevices, AIRI
Abstract: Text-to-image generation models have gained popularity among users around the world. However, many of these models exhibit a strong bias toward English-speaking cultures, ignoring or misrepresenting the unique characteristics of other language groups, countries, and nationalities. The lack of cultural awareness can reduce the generation quality and lead to undesirable consequences such as unintentional insult, and the spread of prejudice. In contrast to the field of natural language processing, cultural awareness in computer vision has not been explored as extensively. In this paper, we strive to reduce this gap. We propose a RusCode benchmark for evaluating the quality of text-to-image generation containing elements of the Russian cultural code. To do this, we form a list of 19 categories that best represent the features of Russian visual culture. Our final dataset consists of 1250 text prompts in Russian and their translations into English. The prompts cover a wide range of topics, including complex concepts from art, popular culture, folk traditions, famous people's names, natural objects, scientific achievements, etc. We present the results of a human evaluation of the side-by-side comparison of Russian visual concepts representations using popular generative models.
Dataset: Russian Jeopardy
Description: A dataset of 29,375 Russian quiz-show questions of the Jeopardy! type, collected from the ChGK database. The questions are fact-oriented and suitable for open-domain question answering tasks.
Language: ru
Domain: TV Channels
Form: text
Size: 29375 sentences
Provider: Tyumen State University
Tasks: question answering
License: custom
Year: 2024
Authors: Elena Mikhalkova, Alexander Khlyupin
Affiliations: Tyumen State University
Abstract: Question answering (QA) is one of the most common NLP tasks that relates to named entity recognition, fact extraction, semantic search and some other fields. In industry, it is much appreciated in chatbots and corporate information systems. It is also a challenging task that attracted the attention of a very general audience at the quiz show Jeopardy! In this article we describe a Jeopardy!-like Russian QA data set collected from the official Russian quiz database Chgk (che ge ka). The data set includes 379,284 quiz-like questions with 29,375 from the Russian analogue of Jeopardy! - "Own Game". We observe its linguistic features and the related QA-task. We conclude about perspectives of a QA competition based on the data set collected from this database.
Dataset: Russian Multimodal Summarization
Description: A multimodal dataset of 420 Russian-language scientific papers from 7 domains (Economics, History, IT, Journalism, Law, Linguistics, Medicine). The dataset includes texts, abstracts, tables, and figures, designed for the task of automatic text summarization.
Language: ru
Domain: web pages
Form: text
Size: 420 documents
Provider: Novosibirsk State University, A.P. Ershov Institute of Informatics Systems
Tasks: summarization
License: unknown
Year: 2024
Authors: Tsanda Alena, Bruches Elena
Affiliations: Novosibirsk State University, A.P. Ershov Institute of Informatics Systems
Abstract: The paper discusses the creation of a multimodal dataset of Russian-language scientific papers and testing of existing language models for the task of automatic text summarization. A feature of the dataset is its multimodal data, which includes texts, tables and figures. The paper presents the results of experiments with two language models: Gigachat from SBER and YandexGPT from Yandex. The dataset consists of 420 papers and is publicly available on https://github.com/iis-research-team/summarization-dataset.
Dataset: NEREL
Description: NEREL is a Russian dataset for named entity recognition and relation extraction. It contains 56K annotated named entities and 39K annotated relations.
Language: ru
Domain: news articles, wikipedia
Form: text
Size: 56000 tokens
Provider: HSE University
Tasks: named entity recognition, relation extraction
License: unknown
Year: 2021
Authors: Natalia Loukachevitch, Ekaterina Artemova, Tatiana Batura, Pavel Braslavski, Ilia Denisov, Vladimir Ivanov, Suresh Manandhar, Alexander Pugachev, Elena Tutubalina
Affiliations: Lomonosov Moscow State University, HSE University, Huawei Noah's Ark lab, Novosibirsk State University, Ural Federal University, Innopolis University, Kazan Federal University, Sber AI, Wiseyak
Abstract: In this paper, we present NEREL, a Russian dataset for named entity recognition and relation extraction. NEREL is significantly larger than existing Russian datasets: to date it contains 56K annotated named entities and 39K annotated relations. Its important difference from previous datasets is annotation of nested named entities, as well as relations within nested entities and at the discourse level. NEREL can facilitate development of novel models that can extract relations between nested named entities, as well as relations on both sentence and document levels. NEREL also contains the annotation of events involving named entities and their roles in the events. The NEREL collection is available via https://github.com/nerel-ds/NEREL.
Dataset: HISTOIRESMORALES
Description: A French dataset for assessing moral alignment in LLMs, derived from the English MORALSTORIES dataset. It consists of 12,000 short narratives describing social situations, moral norms, intentions, and corresponding moral/immoral actions and consequences, adapted to the French cultural context.
Language: fr
Domain: public datasets
Form: text
Size: 12000 sentences
Provider: Laboratoire Hubert Curien
Tasks: topic classification, natural language inference, linguistic acceptability
License: MIT License
Year: 2025
Authors: Thibaud Leteno, Irina Proskurina, Antoine Gourru, Julien Velcin, Charlotte Laclau, Guillaume Metzler, Christophe Gravier
Affiliations: Laboratoire Hubert Curien, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, ERIC, Télécom Paris, Institut Polytechnique de Paris
Abstract: Aligning language models with human values is crucial, especially as they become more integrated into everyday life. While models are often adapted to user preferences, it is equally important to ensure they align with moral norms and behaviours in real-world social situations. Despite significant progress in languages like English and Chinese, French has seen little attention in this area, leaving a gap in understanding how LLMs handle moral reasoning in this language. To address this gap, we introduce Histoires Morales, a French dataset derived from Moral Stories, created through translation and subsequently refined with the assistance of native speakers to guarantee grammatical accuracy and adaptation to the French cultural context. We also rely on annotations of the moral values within the dataset to ensure their alignment with French norms. Histoires Morales covers a wide range of social situations, including differences in tipping practices, expressions of honesty in relationships, and responsibilities toward animals. To foster future research, we also conduct preliminary experiments on the alignment of multilingual models on French and English data and the robustness of the alignment. We find that while LLMs are generally aligned with human moral norms by default, they can be easily influenced with user-preference optimization for both moral and immoral data.
Dataset: PxSLU
Description: A spoken medical drug prescription corpus in French, containing about 4 hours of transcribed and annotated dialogues. The data was acquired from 55 expert and non-expert participants interacting with a dialogue system on a smartphone.
Language: fr
Domain: books
Form: audio
Size: 4 hours
Provider: Univ. Grenoble Alpes
Tasks: speech recognition, other
License: CC BY 4.0
Year: 2022
Authors: AliCan Kocabiyikoglu, François Portet, Prudence Gibert, Hervé Blanchon, Jean-Marc Babouchkine, Gaëtan Gavazzi
Affiliations: Univ. Grenoble Alpes, CHU Grenoble Alpes, Calystene SA, Clinique de médecine gériatrique
Abstract: Spoken medical dialogue systems are increasingly attracting interest to enhance access to healthcare services and improve quality and traceability of patient care. In this paper, we focus on medical drug prescriptions acquired on smartphones through spoken dialogue. Such systems would facilitate the traceability of care and would free clinicians' time. However, there is a lack of speech corpora to develop such systems since most of the related corpora are in text form and in English. To facilitate the research and development of spoken medical dialogue systems, we present, to the best of our knowledge, the first spoken medical drug prescriptions corpus, named PxSLU. It contains 4 hours of transcribed and annotated dialogues of drug prescriptions in French acquired through an experiment with 55 participants experts and non-experts in prescriptions. We also present some experiments that demonstrate the interest of this corpus for the evaluation and development of medical dialogue systems.
Dataset: French COVID19 Lockdown Twitter Dataset
Description: A multilingual Twitter dataset, primarily in French, collected during the first COVID-19 lockdown in France starting March 17, 2020. The dataset, collected via Twitter's API using #ConfinementJourXx hashtags, includes sentiment and emotion annotations derived from various lexicons.
Language: multilingual
Domain: social media
Form: text
Size: 2474086 sentences
Provider: University of Picardie, Paris-Nanterre University, Lille University
Tasks: sentiment analysis
License: CC BY-NC-SA 4.0
Year: 2020
Authors: Sophie Balech, Christophe Benavent, Mihai Calciu
Affiliations: University of Picardie, Paris-Nanterre University, Lille University
Abstract: In this paper, we present a mainly French coronavirus Twitter dataset that we have been continuously collecting since lockdown restrictions have been enacted in France (in March 17, 2020). We offer our datasets and sentiment analysis annotations to the research community at https://github.com/calciu/COVID19-LockdownFr. They have been obtained using high performance computing (HPC) capabilities of our university's datacenter. We think that our contribution can facilitate analysis of online conversation dynamics reflecting people sentiments when facing severe home confinement restrictions determined by the outbreak of this world wide epidemic. We hope that our contribution will help decode shared experience and mood but also test the sensitivity of sentiment measurement instruments and incite the development of new instruments, methods and approaches.
Dataset: MEDIA with Intents
Description: An enhanced version of the French MEDIA Spoken Language Understanding (SLU) dataset. Originally annotated only for slots (named entities), this version adds intent classification labels using a semi-automatic tri-training methodology. The corpus consists of recorded human-machine dialogues for hotel booking.
Language: fr
Domain: public datasets
Form: audio
Size: 70 hours
Provider: Université Paris-Saclay, Avignon Université, LIA, QWANT
Tasks: speech recognition, intent classification
License: unknown
Year: 2024
Authors: Nadège Alavoine, Gaëlle Laperrière, Christophe Servan, Sahar Ghannay, Sophie Rosset
Affiliations: Université Paris-Saclay, Avignon Université, QWANT
Abstract: Intent classification and slot-filling are essential tasks of Spoken Language Understanding (SLU). In most SLUsystems, those tasks are realized by independent modules. For about fifteen years, models achieving both of themjointly and exploiting their mutual enhancement have been proposed. A multilingual module using a joint modelwas envisioned to create a touristic dialogue system for a European project, HumanE-AI-Net. A combination ofmultiple datasets, including the MEDIA dataset, was suggested for training this joint model. The MEDIA SLU datasetis a French dataset distributed since 2005 by ELRA, mainly used by the French research community and free foracademic research since 2020. Unfortunately, it is annotated only in slots but not intents. An enhanced version ofMEDIA annotated with intents has been built to extend its use to more tasks and use cases. This paper presents thesemi-automatic methodology used to obtain this enhanced version. In addition, we present the first results of SLUexperiments on this enhanced dataset using joint models for intent classification and slot-filling.
Dataset: FrenchMedMCQA
Description: FrenchMedMCQA is a multiple-choice question answering dataset in French for the medical domain. It contains 3,105 questions taken from real exams of the French medical specialization diploma in pharmacy, mixing single and multiple answers.
Language: fr
Domain: web pages
Form: text
Size: 3105 sentences
Provider: Avignon University, Nantes University, Zenidoc
Tasks: multiple choice question answering
License: Apache-2.0
Year: 2023
Authors: Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, Mickael Rouvier
Affiliations: LIA - Avignon University, LS2N - Nantes University, CHU de Nantes - La clinique des données - Nantes University, Zenidoc
Abstract: This paper introduces FrenchMedMCQA, the first publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain. It is composed of 3,105 questions taken from real exams of the French medical specialization diploma in pharmacy, mixing single and multiple answers. Each instance of the dataset contains an identifier, a question, five possible answers and their manual correction(s). We also propose first baseline models to automatically process this MCQA task in order to report on the current performances and to highlight the difficulty of the task. A detailed analysis of the results showed that it is necessary to have representations adapted to the medical domain or to the MCQA task: in our case, English specialized models yielded better results than generic French ones, even though FrenchMedMCQA is in French. Corpus, models and tools are available online.
Dataset: FRASIMED
Description: A French annotated corpus comprising 2,051 synthetic clinical cases for Named Entity Recognition (NER) and Entity Linking. It was created by projecting annotations from two Spanish corpora, CANTEMIST and DISTEMIST, using a crosslingual BERT-based method, followed by manual correction. The annotations are linked to medical terminologies like ICD-O and SNOMED-CT.
Language: fr
Domain: public datasets
Form: text
Size: 24037 tokens
Provider: University Hospitals of Geneva, University of Geneva
Tasks: named entity recognition
License: CC BY 4.0
Year: 2024
Authors: Jamil Zaghir, Mina Bjelogrlic, Jean-Philippe Goldman, Soukaïna Aananou, Christophe Gaudet-Blavignac, Christian Lovis
Affiliations: Division of Medical Information Sciences (SIMED), University Hospitals of Geneva, Geneva, Switzerland, Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
Abstract: Natural language processing (NLP) applications such as named entity recognition (NER) for low-resource corpora do not benefit from recent advances in the development of large language models (LLMs) where there is still a need for larger annotated datasets. This research article introduces a methodology for generating translated versions of annotated datasets through crosslingual annotation projection. Leveraging a language agnostic BERT-based approach, it is an efficient solution to increase low-resource corpora with few human efforts and by only using already available open data resources. Quantitative and qualitative evaluations are often lacking when it comes to evaluating the quality and effectiveness of semi-automatic data generation strategies. The evaluation of our crosslingual annotation projection approach showed both effectiveness and high accuracy in the resulting dataset. As a practical application of this methodology, we present the creation of French Annotated Resource with Semantic Information for Medical Entities Detection (FRASIMED), an annotated corpus comprising 2'051 synthetic clinical cases in French. The corpus is now available for researchers and practitioners to develop and refine French natural language processing (NLP) applications in the clinical field (https://zenodo.org/record/8355629), making it the largest open annotated corpus with linked medical concepts in French.
Dataset: FQuAD1.1
Description: FQuAD is a French Question Answering dataset composed of 10,000 question-answer pairs extracted from news articles. The dataset is designed to be used for training and evaluating question answering models in French.
Language: fr
Domain: wikipedia
Form: text
Size: 62003 sentences
Provider: Illuin Technology
Tasks: question answering
License: CC BY-NC-SA 3.0
Year: 2020
Authors: Martin d'Hoffschmidt, Wacim Belblidia, Tom Brendlé, Quentin Heinrich, Maxime Vidal
Affiliations: Illuin Technology, ETH Zurich
Abstract: Recent advances in the field of language modeling have improved state-of-the-art results on many Natural Language Processing tasks. Among them, Reading Comprehension has made significant progress over the past few years. However, most results are reported in English since labeled resources available in other languages, such as French, remain scarce. In the present work, we introduce the French Question Answering Dataset (FQuAD). FQuAD is a French Native Reading Comprehension dataset of questions and answers on a set of Wikipedia articles that consists of 25,000+ samples for the 1.0 version and 60,000+ samples for the 1.1 version. We train a baseline model which achieves an F1 score of 92.2 and an exact match ratio of 82.1 on the test set. In order to track the progress of French Question Answering models we propose a leader-board and we have made the 1.0 version of our dataset freely available at https://illuin-tech.github.io/FQuAD-explorer/.
Dataset: FIJO
Description: FIJO (French Insurance Job Offer) is a public dataset containing 867 French job offers from the insurance domain. A subset of 47 job ads is annotated with four types of soft skills (Thoughts, Results, Relational, Personal). The dataset was created to facilitate research in automatic skill recognition.
Language: fr
Domain: web pages
Form: text
Size: 932 tokens
Provider: Université Laval
Tasks: named entity recognition
License: CC BY-NC-SA 4.0
Year: 2022
Authors: David Beauchemin, Julien Laumonier, Yvan Le Ster, Marouane Yassine
Affiliations: Departement of Computer Science and Software Engineering, Université Laval, Québec, Canada, Institute Intelligence and Data, Université Laval, Québec, Canada
Abstract: Understanding the evolution of job requirements is becoming more important for workers, companies and public organizations to follow the fast transformation of the employment market. Fortunately, recent natural language processing (NLP) approaches allow for the development of methods to automatically extract information from job ads and recognize skills more precisely. However, these efficient approaches need a large amount of annotated data from the studied domain which is difficult to access, mainly due to intellectual property. This article proposes a new public dataset, FIJO, containing insurance job offers, including many soft skill annotations. To understand the potential of this dataset, we detail some characteristics and some limitations. Then, we present the results of skill detection algorithms using a named entity recognition approach and show that transformers-based models have good token-wise performances on this dataset. Lastly, we analyze some errors made by our best model to emphasize the difficulties that may arise when applying NLP approaches.
Dataset: BSARD
Description: A dataset of French legal questions posed by Belgian citizens and labelled with relevant articles from the Belgian legislation.
Language: fr
Domain: books
Form: text
Size: 1108 sentences
Provider: Maastricht University
Tasks: information retrieval, question answering
License: CC BY-NC-SA 4.0
Year: 2022
Authors: Antoine Louis, Gerasimos Spanakis
Affiliations: Maastricht University
Abstract: Statutory article retrieval is the task of automatically retrieving law articles relevant to a legal question. While recent advances in natural language processing have sparked considerable interest in many legal tasks, statutory article retrieval remains primarily untouched due to the scarcity of large-scale and high-quality annotated datasets. To address this bottleneck, we introduce the Belgian Statutory Article Retrieval Dataset (BSARD), which consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles. Using BSARD, we benchmark several state-of-the-art retrieval approaches, including lexical and dense architectures, both in zero-shot and supervised setups. We find that fine-tuned dense retrieval models significantly outperform other systems. Our best performing baseline achieves 74.8% R@100, which is promising for the feasibility of the task and indicates there is still room for improvement. By the specificity of the domain and addressed task, BSARD presents a unique challenge problem for future research on legal information retrieval. Our dataset and source code are publicly available.
Dataset: FairTranslate
Description: An English-French dataset for evaluating non-binary gender bias in machine translation. It contains 2,418 human-annotated sentence pairs about occupations, with metadata for gender (male, female, inclusive), stereotype alignment, and ambiguity, designed to test LLMs' handling of inclusive language like the singular 'they'.
Language: multilingual
Domain: LLM
Form: text
Size: 2418 sentences
Provider: IRT Saint Exupéry
Tasks: machine translation, gender identification
License: CC BY 4.0
Year: 2025
Authors: Fanny Jourdan, Yannick Chevalier, Cécile Favre
Affiliations: IRT Saint Exupery, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, ERIC
Abstract: Large Language Models (LLMs) are increasingly leveraged for translation tasks but often fall short when translating inclusive language -- such as texts containing the singular 'they' pronoun or otherwise reflecting fair linguistic protocols. Because these challenges span both computational and societal domains, it is imperative to critically evaluate how well LLMs handle inclusive translation with a well-founded framework.   This paper presents FairTranslate, a novel, fully human-annotated dataset designed to evaluate non-binary gender biases in machine translation systems from English to French. FairTranslate includes 2418 English-French sentence pairs related to occupations, annotated with rich metadata such as the stereotypical alignment of the occupation, grammatical gender indicator ambiguity, and the ground-truth gender label (male, female, or inclusive).   We evaluate four leading LLMs (Gemma2-2B, Mistral-7B, Llama3.1-8B, Llama3.3-70B) on this dataset under different prompting procedures. Our results reveal substantial biases in gender representation across LLMs, highlighting persistent challenges in achieving equitable outcomes in machine translation. These findings underscore the need for focused strategies and interventions aimed at ensuring fair and inclusive language usage in LLM-based translation systems.   We make the FairTranslate dataset publicly available on Hugging Face, and disclose the code for all experiments on GitHub.