IVUL-KAUST/MOLE huggingface dataset:
Dataset: GoEmotions
Description: A large-scale, manually annotated dataset of 58,009 English Reddit comments. The comments are labeled for 27 fine-grained emotion categories or Neutral, designed for emotion classification and understanding tasks. The dataset was curated to balance sentiment and reduce profanity and harmful content.
Language: en
Domain: social media
Form: text
Size: 58009 sentences
Provider: Google Research
Tasks: emotion classification
License: Apache-2.0
Year: 2020
Authors: Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, Sujith Ravi
Affiliations: Stanford Linguistics, Google Research, Amazon Alexa
Abstract: Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks. We introduce GoEmotions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demonstrate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with existing emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT-based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.
Dataset: SQuAD 2.0
Description: A version of the Stanford Question Answering Dataset (SQuAD) that combines existing SQuAD 1.1 data with over 50,000 new, unanswerable questions written adversarially by crowdworkers.
Language: en
Domain: wikipedia
Form: text
Size: 151054 sentences
Provider: Stanford University
Tasks: question answering
License: CC BY-SA 4.0
Year: 2018
Authors: Pranav Rajpurkar, Robin Jia, Percy Liang
Affiliations: Stanford University
Abstract: Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.
Dataset: LAMBADA
Description: A dataset of narrative passages designed for a word prediction task. The key characteristic is that human subjects can easily guess the final word of a passage when given the full context, but find it nearly impossible when only shown the last sentence.
Language: en
Domain: books, public datasets
Form: text
Size: 10022 documents
Provider: University of Trento, University of Amsterdam
Tasks: word prediction
License: unknown
Year: 2016
Authors: Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, Raquel Fernández
Affiliations: University of Trento, University of Amsterdam
Abstract: We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.
Dataset: ClimbMix
Description: ClimbMix is a compact 400-billion-token dataset designed for efficient language model pre-training. It was created by applying the CLIMB framework to find an optimal mixture from the ClimbLab corpus (derived from Nemotron-CC and smollm-corpus), delivering superior performance under an equal token budget.
Language: en
Domain: web pages, LLM, other
Form: text
Size: 400000000000 tokens
Provider: NVIDIA
Tasks: language modeling
License: CC BY-NC 4.0
Year: 2025
Authors: Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan (Celine) Lin, Jan Kautz, Pavlo Molchanov
Affiliations: NVIDIA
Abstract: Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/
Dataset: RACE
Description: A large-scale reading comprehension dataset collected from English exams for middle and high school Chinese students. It consists of nearly 28,000 passages and 100,000 multiple-choice questions designed by human experts to evaluate understanding and reasoning abilities, covering a variety of topics.
Language: en
Domain: web pages
Form: text
Size: 97687 sentences
Provider: Carnegie Mellon University
Tasks: multiple choice question answering
License: custom
Year: 2017
Authors: Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy
Affiliations: Carnegie Mellon University
Abstract: We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.
Dataset: MMLU-Pro
Description: An enhanced version of the MMLU benchmark, MMLU-Pro features more challenging, reasoning-focused questions with an expanded choice set of ten options. It was created by filtering trivial and noisy questions from MMLU and integrating new questions, followed by expert review, to be more discriminative and robust.
Language: en
Domain: public datasets, web pages, LLM
Form: text
Size: 12032 sentences
Provider: University of Waterloo, University of Toronto, Carnegie Mellon University
Tasks: multiple choice question answering
License: MIT License
Year: 2024
Authors: Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Wenhu Chen
Affiliations: University of Waterloo, University of Toronto, Carnegie Mellon University
Abstract: In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.
Dataset: BoolQ
Description: A reading comprehension dataset of 16,000 naturally occurring yes/no questions. Questions are gathered from unprompted Google search queries and paired with a Wikipedia paragraph containing the answer. The dataset is designed to be challenging, requiring complex, non-factoid inference.
Language: en
Domain: wikipedia, web pages
Form: text
Size: 16000 sentences
Provider: Google AI
Tasks: question answering
License: CC BY-SA 3.0
Year: 2019
Authors: Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova
Affiliations: Paul G. Allen School of CSE, University of Washington, Google AI Language
Abstract: In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
Dataset: GSM8K
Description: GSM8K is a dataset of 8.5K high quality grade school math problems created by human problem writers. The dataset is designed to have high linguistic diversity while relying on relatively simple grade school math concepts.
Language: en
Domain: other
Form: text
Size: 8500 sentences
Provider: OpenAI
Tasks: question answering
License: MIT License
Year: 2021
Authors: Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Łukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman
Affiliations: OpenAI
Abstract: State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Dataset: HotpotQA
Description: A large-scale dataset with 113k Wikipedia-based question-answer pairs. It requires reasoning over multiple supporting documents, features diverse questions, provides sentence-level supporting facts for explainability, and includes factoid comparison questions to test systems' ability to extract and compare facts.
Language: en
Domain: wikipedia
Form: text
Size: 112779 sentences
Provider: Carnegie Mellon University, Stanford University, Mila, Université de Montréal, Google AI
Tasks: question answering
License: CC BY-SA 4.0
Year: 2018
Authors: Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning
Affiliations: Carnegie Mellon University, Stanford University, Mila, Université de Montréal, Google AI
Abstract: Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.
Dataset: SQuAD
Description: A reading comprehension dataset consisting of over 100,000 questions posed by crowdworkers on a set of Wikipedia articles. The answer to each question is a segment of text, or span, from the corresponding reading passage.
Language: en
Domain: wikipedia
Form: text
Size: 107785 sentences
Provider: Stanford University
Tasks: question answering
License: CC BY-SA 4.0
Year: 2016
Authors: Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang
Affiliations: Stanford University
Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research.   The dataset is freely available at https://stanford-qa.com
Dataset: RefinedWeb
Description: A large-scale, five trillion token English pretraining dataset derived from CommonCrawl. It was created using extensive filtering and deduplication to demonstrate that high-quality web data alone can produce models that outperform those trained on curated corpora. A 600 billion token extract is publicly available.
Language: en
Domain: web pages
Form: text
Size: 600000000000 tokens
Provider: Technology Innovation Institute
Tasks: language modeling
License: ODC-By
Year: 2023
Authors: Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay
Affiliations: LightOn, Technology Innovation Institute, LPENS, École normale supérieure
Abstract: Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
Dataset: MMLU
Description: The MMLU dataset is a collection of 57 tasks covering a wide range of subjects, including elementary mathematics, US history, computer science, law, and more. The dataset is designed to measure a text model's multitask accuracy and requires models to possess extensive world knowledge and problem-solving ability.
Language: en
Domain: web pages, books
Form: text
Size: 15908 sentences
Provider: UC Berkeley, Columbia University, UChicago, UIUC
Tasks: multiple choice question answering
License: MIT License
Year: 2021
Authors: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt
Affiliations: UC Berkeley, Columbia University, UChicago, UIUC
Abstract: We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
Dataset: PIQA
Description: A benchmark dataset for physical commonsense reasoning, presented as multiple-choice question answering. It contains goal-solution pairs inspired by how-to instructions from instructables.com, designed to test a model's understanding of physical properties, affordances, and object manipulation. The dataset was cleaned of artifacts using the AFLite algorithm.
Language: en
Domain: web pages
Form: text
Size: 21000 sentences
Provider: Allen Institute for Artificial Intelligence, Microsoft Research AI, Carnegie Mellon University, University of Washington
Tasks: question answering, multiple choice question answering, commonsense reasoning
License: AFL-3.0
Year: 2019
Authors: Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi
Affiliations: Allen Institute for Artificial Intelligence, Microsoft Research AI, Carnegie Mellon University, Paul G. Allen School for Computer Science and Engineering, University of Washington
Abstract: To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.