Omkar2118/datasets huggingface dataset:
GENIE
Full name: GENeratIve Evaluation

GENIE, which stands for GENeratIve Evaluation, is a system designed to standardize human evaluations across different text generation tasks. It was introduced to produce consistent evaluations that are reproducible over time and across different populations. The system is instantiated with datasets representing four core challenges in text generation: machine translation, summarization, commonsense reasoning, and machine comprehension. For each task, GENIE offers a leaderboard that automatically crowdsources annotations for submissions, evaluating them along axes such as correctness, conciseness, and fluency.

Introducing paper: GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
SummScreen

SummScreen is a dataset for abstractive screenplay summarization. It consists of pairs of TV series transcripts and human-written recaps. This dataset provides a challenging testbed for abstractive summarization for several reasons:
- Plot details are often expressed indirectly in character dialogues and may be scattered across the entirety of the transcript.
- These details must be found and integrated to form the succinct plot descriptions in the recaps.
- TV scripts contain content that does not directly pertain to the central plot but rather serves to develop characters or provide comic relief. This information is rarely contained in recaps.

Introducing paper: SummScreen: A Dataset for Abstractive Screenplay Summarization
Bamboogle

The Bamboogle dataset is a collection of questions that was constructed to investigate the ability of language models to perform compositional reasoning tasks. The dataset is made up of questions that Google answers incorrectly. It covers many different types of questions on various areas, written in unique ways.

Introducing paper: Measuring and Narrowing the Compositionality Gap in Language Models
classification benchmark

This benchmark includes 11 image classification datasets that were used to evaluate the transferability of metrics. Datasets include FGVC Aircraft, Caltech101, Stanford Cars, CIFAR-10, CIFAR-100, DTD, Oxford-102, Flowers, Food-101, Oxford-IIIT Pets, SUN397, and VOC2007 . Please refer to SFDA (https://github.com/TencentARC/SFDA) or ETran (https://github.com/mgholamikn/ETran/tree/main) for further details about the benchmark.
ARAS
Full name: Action with RAre Scene

Action with RAre Scene is a small scale dataset collected from Youtube. By definition, it includes video clips of human actions (those action categories fall into Kinetics-400 action classes) with rare scenes or backgrounds.

Introducing paper: Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarks
MVBench

MVBench is a comprehensive Multi-modal Video understanding Benchmark. It was introduced to evaluate the comprehension capabilities of Multi-modal Large Language Models (MLLMs), particularly their temporal understanding in dynamic video tasks. MVBench covers 20 challenging video tasks that cannot be effectively solved with a single frame. It introduces a novel static-to-dynamic method to define these temporal-related tasks. By transforming various static tasks into dynamic ones, it enables the systematic generation of video tasks that require a broad spectrum of temporal skills, ranging from perception to cognition.

Introducing paper: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
CommonMT

The CommonMT is a dataset used for evaluating commonsense reasoning in neural machine translation. This dataset contains three types of test suites:
- Lexical ambiguity
- Contextless syntactic ambiguity
- Contextual syntactic ambiguity
GLUE-X

GLUE-X is a benchmark dataset used to evaluate the out-of-distribution (OOD) robustness of Natural Language Understanding (NLU) models. It was created to address the OOD generalization problem, which remains a challenge in many NLP tasks and limits the real-world deployment of these methods. The GLUE-X dataset consists of 14 publicly available datasets used as OOD test data. Evaluations are conducted on 8 classic NLP tasks over popularly used models. The findings from these evaluations highlight the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy. The creators of GLUE-X hope that this dataset will help highlight the importance of OOD robustness and provide insights on how to measure the robustness of a model and how to improve it.

Introducing paper: GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
ePiC

The ePiC dataset is a unique and high-quality crowdsourced collection of narratives specifically designed for testing abstract language understanding in the context of proverbs. The ePiC dataset stands out for its focus on abstract reasoning in language models, a relatively under-explored area in natural language processing.

Introducing paper: ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
characterRelations

characterRelations dataset contains 2,170 annotations of character relations in 109 literary texts, as documented in characterRelations.pdf. Each annotation describes a character dyad along four dimensions of interest: coarse-grained category (social, familial, professional), fine-grained category (e.g., friend, lover, parent, rival, employer), and affinity (positive, negative, neutral). Additionally, we do not assume that this relationship is static; we also collect judgments as to whether it changes at any point in the course of the text.

Introducing paper: Annotating Character Relationships in Literary Texts
ISNotes

The ISNotes dataset is a corpus used for fine-grained Information Status (IS) classification. IS reflects the accessibility of a discourse entity based on the evolving discourse context and the speaker’s assumption about the hearer’s knowledge and beliefs. According to Markert et al. (2012), old mentions refer to entities that have been referred to previously; mediated mentions have not been mentioned before but are accessible to the hearer by reference to another old mention or to prior world knowledge; and new mentions refer to entities that are introduced to the discourse for the first time and are not known to the hearer before.

Introducing paper: Collective Classification for Fine-grained Information Status
BASHI

The BASHI dataset is a corpus consisting of 50 Wall Street Journal (WSJ) articles. It adds bridging anaphors and their antecedents to the other gold annotations that have been created as part of the OntoNotes project. Bridging anaphors are context-dependent expressions that do not refer to the same entity as their antecedent but to a related entity.

Introducing paper: BASHI: A Corpus of Wall Street Journal Articles Annotated with Bridging Links