IVUL-KAUST/MOLE huggingface dataset:
Dataset: BRIGHT
Description: BRIGHT is a new benchmark for reasoning-intensive retrieval. It consists of 12 datasets from diverse and advanced domains where relevance between queries and documents requires intensive reasoning beyond simple keyword or semantic matching.
Language: en
Domain: web pages, public datasets, LLM
Form: text
Size: 1384 sentences
Provider: The University of Hong Kong, Princeton University, Stanford University, University of Washington, Google Cloud AI Research
Tasks: information retrieval, question answering
License: CC BY 4.0
Year: 2025
Authors: Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arık, Danqi Chen, Tao Yu
Affiliations: The University of Hong Kong, Princeton University, Stanford University, University of Washington, Google Cloud AI Research
Abstract: Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.
Dataset: HLE
Description: Humanity's Last Exam (HLE) is a dataset of 3,000 challenging questions designed to assess the capabilities of large language models (LLMs). The questions are diverse, covering a wide range of topics and requiring different reasoning abilities. The dataset is still under development and accepting new questions.
Language: en
Domain: other
Form: text
Size: 2500 sentences
Provider: Center for AI Safety, Scale AI
Tasks: question answering, multiple choice question answering
License: MIT License
Year: 2025
Authors: Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Summer Yue, Alexandr Wang, Dan Hendrycks
Affiliations: Center for AI Safety, Scale AI
Abstract: Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
Dataset: TinyStories
Description: TinyStories is a synthetic dataset of short stories generated by GPT-3.5 and GPT-4. The stories are designed to be simple, using only words that a typical 3 to 4-year-old understands. It is intended to train and evaluate small language models on their ability to generate coherent text and demonstrate reasoning.
Language: en
Domain: LLM
Form: text
Size: 2141709 documents
Provider: Microsoft Research
Tasks: language modeling, text generation, instruction tuning
License: CDLA-SHARING-1.0
Year: 2023
Authors: Ronen Eldan, Yuanzhi Li
Affiliations: Microsoft Research
Abstract: Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention).   In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.   We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency.   We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.
Dataset: WinoGrande
Description: WinoGrande is a large-scale dataset of 44,000 problems inspired by the original Winograd Schema Challenge (WSC). The dataset was constructed through a carefully designed crowdsourcing procedure followed by a systematic bias reduction.
Language: en
Domain: web pages
Form: text
Size: 43972 sentences
Provider: Allen Institute for Artificial Intelligence, University of Washington
Tasks: commonsense reasoning
License: CC BY 4.0
Year: 2019
Authors: Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi
Affiliations: Allen Institute for Artificial Intelligence, University of Washington
Abstract: The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. Furthermore, we establish new state-of-the-art results on five related benchmarks - WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.
Dataset: SciQ
Description: SciQ is a dataset containing 13,679 multiple-choice science exam questions. It was created using a novel crowdsourcing method that leverages a large corpus of domain-specific text (science textbooks) and a model trained on existing questions to suggest document selections and answer distractors, aiding human workers in the question generation process.
Language: en
Domain: books, web pages
Form: text
Size: 13679 sentences
Provider: Allen Institute for Artificial Intelligence
Tasks: multiple choice question answering, question answering
License: CC BY-NC 3.0
Year: 2017
Authors: Johannes Welbl, Nelson F. Liu, Matt Gardner
Affiliations: Allen Institute for Artificial Intelligence, University of Washington, University College London
Abstract: We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions (Dataset available at http://allenai.org/data.html). We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.
Dataset: TriviaQA
Description: TriviaQA is a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions.
Language: en
Domain: wikipedia, web pages
Form: text
Size: 650000 documents
Provider: University of Washington
Tasks: question answering, information retrieval
License: Apache-2.0
Year: 2017
Authors: Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer
Affiliations: Allen Institute for Artificial Intelligence, University of Washington
Abstract: We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study. Data and code available at -- http://nlp.cs.washington.edu/triviaqa/
Dataset: JEC
Description: A quality estimation (QE) dataset created for building an automatic evaluation model for Japanese Grammatical Error Correction (GEC).
Language: jp
Script: mixed
Domain: public datasets
Form: text
Size: 4391 sentences
Provider: Tokyo Metropolitan University, RIKEN
Tasks: grammatical error correction
License: unknown
Year: 2022
Authors: Daisuke Suzuki, Yujin Takahashi, Ikumi Yamashita, Taichi Aida, Tosho Hirasawa, Michitaka Nakatsuji, Masato Mita, Mamoru Komachi
Affiliations: Tokyo Metropolitan University, RIKEN
Abstract: In grammatical error correction (GEC), automatic evaluation is an important factor for research and development of GEC systems. Previous studies on automatic evaluation have demonstrated that quality estimation models built from datasets with manual evaluation can achieve high performance in automatic evaluation of English GEC without using reference sentences.. However, quality estimation models have not yet been studied in Japanese, because there are no datasets for constructing quality estimation models. Therefore, in this study, we created a quality estimation dataset with manual evaluation to build an automatic evaluation model for Japanese GEC. Moreover, we conducted a meta-evaluation to verify the dataset's usefulness in building the Japanese quality estimation model.
Dataset: JParaCrawl
Description: JParaCrawl is a large web-based English-Japanese parallel corpus that was created by crawling the web and finding English-Japanese bitexts. It contains around 8.7 million parallel sentences.
Language: multilingual
Script: mixed
Domain: web pages
Form: text
Size: 8763995 sentences
Provider: NTT
Tasks: machine translation
License: custom
Year: 2020
Authors: Makoto Morishita, Jun Suzuki, Masaaki Nagata
Affiliations: NTT Corporation
Abstract: Recent machine translation algorithms mainly rely on parallel corpora. However, since the availability of parallel corpora remains limited, only some resource-rich language pairs can benefit from them. We constructed a parallel corpus for English-Japanese, for which the amount of publicly available parallel corpora is still limited. We constructed the parallel corpus by broadly crawling the web and automatically aligning parallel sentences. Our collected corpus, called JParaCrawl, amassed over 8.7 million sentence pairs. We show how it includes a broader range of domains and how a neural machine translation model trained with it works as a good pre-trained model for fine-tuning specific domains. The pre-training and fine-tuning approaches achieved or surpassed performance comparable to model training from the initial state and reduced the training time. Additionally, we trained the model with an in-domain dataset and JParaCrawl to show how we achieved the best performance with them. JParaCrawl and the pre-trained models are freely available online for research purposes.
Dataset: KaoKore
Description: KaoKore is a dataset of 5,552 face images extracted from pre-modern Japanese artwork from the 16th to 17th centuries. It is derived from the 'Collection of Facial Expressions' dataset and provides labels for gender and social status, along with official train/dev/test splits for classification and generative tasks.
Language: jp
Script: mixed
Domain: web pages
Form: images
Size: 5552 images
Provider: Google Brain, ROIS-DS Center for Open Data in the Humanities, NII, University of Cambridge, MILA, Universit'e de Montr'eal
Tasks: gender identification, other
License: CC BY-SA 4.0
Year: 2020
Authors: Yingtao Tian, Chikahiko Suzuki, Tarin Clanuwat, Mikel Bober-Irizar, Alex Lamb, Asanobu Kitamoto
Affiliations: Google Brain, ROIS-DS Center for Open Data in the Humanities, NII, University of Cambridge, MILA, Universit'e de Montr'eal
Abstract: From classifying handwritten digits to generating strings of text, the datasets which have received long-time focus from the machine learning community vary greatly in their subject matter. This has motivated a renewed interest in building datasets which are socially and culturally relevant, so that algorithmic research may have a more direct and immediate impact on society. One such area is in history and the humanities, where better and relevant machine learning models can accelerate research across various fields. To this end, newly released benchmarks and models have been proposed for transcribing historical Japanese cursive writing, yet for the field as a whole using machine learning for historical Japanese artworks still remains largely uncharted. To bridge this gap, in this work we propose a new dataset KaoKore which consists of faces extracted from pre-modern Japanese artwork. We demonstrate its value as both a dataset for image classification as well as a creative and artistic dataset, which we explore using generative models. Dataset available at https://github.com/rois-codh/kaokore
Dataset: llm-japanese-dataset v0
Description: A Japanese chat dataset of approximately 8.4 million records, created for tuning large language models. It is composed of various sub-datasets covering tasks like translation, knowledge-based Q&A, summarization, and more, derived from sources like Wikipedia, WordNet, and other publicly available corpora.
Language: multilingual
Script: mixed
Domain: public datasets, wikipedia, news articles, LLM
Form: text
Size: 8393726 sentences
Provider: The University of Tokyo
Tasks: machine translation, text generation, instruction tuning
License: CC BY-SA 4.0
Year: 2023
Authors: Masanori HIRANO, Masahiro SUZUKI, Hiroki SAKAJI
Affiliations: The University of Tokyo
Abstract: This study constructed a Japanese chat dataset for tuning large language models (LLMs), which consist of about 8.4 million records. Recently, LLMs have been developed and gaining popularity. However, high-performing LLMs are usually mainly for English. There are two ways to support languages other than English by those LLMs: constructing LLMs from scratch or tuning existing models. However, in both ways, datasets are necessary parts. In this study, we focused on supporting Japanese in those LLMs and making a dataset for training or tuning LLMs in Japanese. The dataset we constructed consisted of various tasks, such as translation and knowledge tasks. In our experiment, we tuned an existing LLM using our dataset and evaluated the performance qualitatively. The results suggest that our dataset is possibly beneficial for LLMs. However, we also revealed some difficulties in constructing LLMs in languages other than English.
Dataset: JaLeCoN
Description: JaLeCoN is a Dataset of Japanese Lexical Complexity for Non-Native Readers. It can be used to train or evaluate Japanese lexical complexity prediction models.
Language: jp
Script: mixed
Domain: news articles, public datasets, web pages
Form: text
Size: 600 sentences
Provider: NAIST
Tasks: other
License: CC BY-NC-SA 3.0
Year: 2023
Authors: Yusuke Ide, Masato Mita, Adam Nohejl, Hiroki Ouchi, Taro Watanabe
Affiliations: NAIST, CyberAgent Inc., RIKEN
Abstract: Lexical complexity prediction (LCP) is the task of predicting the complexity of words in a text on a continuous scale. It plays a vital role in simplifying or annotating complex words to assist readers. To study lexical complexity in Japanese, we construct the first Japanese LCP dataset. Our dataset provides separate complexity scores for Chinese/Korean annotators and others to address the readers' L1-specific needs. In the baseline experiment, we demonstrate the effectiveness of a BERT-based system for Japanese LCP.
Dataset: JaQuAD
Description: JaQuAD is a Japanese Question Answering dataset consisting of 39,696 extractive question-answer pairs on Japanese Wikipedia articles. The dataset was annotated by humans and is available on GitHub.
Language: jp
Script: mixed
Domain: wikipedia
Form: text
Size: 39696 sentences
Provider: Skelter Labs
Tasks: question answering
License: CC BY-SA 3.0
Year: 2022
Authors: ByungHoon So, Kyuhong Byun, Kyungwon Kang, Seongjin Cho
Affiliations: Skelter Labs
Abstract: Question Answering (QA) is a task in which a machine understands a given document and a question to find an answer. Despite impressive progress in the NLP area, QA is still a challenging problem, especially for non-English languages due to the lack of annotated datasets. In this paper, we present the Japanese Question Answering Dataset, JaQuAD, which is annotated by humans. JaQuAD consists of 39,696 extractive question-answer pairs on Japanese Wikipedia articles. We finetuned a baseline model which achieves 78.92% for F1 score and 63.38% for EM on test set. The dataset and our experiments are available at https://github.com/SkelterLabsInc/JaQuAD.
Dataset: JAFFE
Description: The Japanese Female Facial Expression (JAFFE) dataset is a set of 213 images depicting facial expressions posed by 10 Japanese women. The set includes six basic facial expressions plus a neutral face. The dataset also includes semantic ratings for each image from 60 Japanese female observers.
Language: jp
Script: mixed
Domain: other
Form: images
Size: 213 images
Provider: Kyushu University, Advanced Telecommunications Research Institute International
Tasks: other
License: custom
Year: 2021
Authors: Michael J. Lyons
Affiliations: Ritsumeikan University
Abstract: Twenty-five years ago, my colleagues Miyuki Kamachi and Jiro Gyoba and I designed and photographed JAFFE, a set of facial expression images intended for use in a study of face perception. In 2019, without seeking permission or informing us, Kate Crawford and Trevor Paglen exhibited JAFFE in two widely publicized art shows. In addition, they published a nonfactual account of the images in the essay "Excavating AI: The Politics of Images in Machine Learning Training Sets." The present article recounts the creation of the JAFFE dataset and unravels each of Crawford and Paglen's fallacious statements. I also discuss JAFFE more broadly in connection with research on facial expression, affective computing, and human-computer interaction.