Omkar2118/datasets huggingface dataset:
HEIM
Full name: Holistic Evaluation of Text-to-Image Models

**HEIM** stands for **Holistic Evaluation of Text-To-Image Models**. It is a comprehensive benchmark designed to assess the capabilities and risks of text-to-image generation models. Unlike previous evaluations that primarily focused on image-text alignment and image quality, HEIM considers **12 different aspects** that are crucial for real-world model deployment:

1. **Image-Text Alignment**
2. **Image Quality**
3. **Aesthetics**
4. **Originality**
5. **Reasoning**
6. **Knowledge**
7. **Bias**
8. **Toxicity**
9. **Fairness**
10. **Robustness**
11. **Multilinguality**
12. **Efficiency**

By curating scenarios that encompass these aspects, HEIM evaluates state-of-the-art text-to-image models. Interestingly, no single model excels in all aspects; different models demonstrate strengths in different areas. For transparency, all prompts, generated images, and results are available on the [HEIM website](https://crfm.stanford.edu/helm/heim/latest/) for exploration and study. Additionally, the [GitHub repository](https://github.com/stanford-crfm/helm) provides a collection of models accessible via a unified API, along with metrics beyond accuracy, such as efficiency, bias, and toxicity.

Introducing paper: Holistic Evaluation of Text-To-Image Models
SuperBench

SuperBench is a comprehensive evaluation system for large language models that includes five benchmark datasets: ExtremeGLUE for semantics, CodeBench for code, AlignBench for alignment, AgentBench for intelligent agents, and SafetyBench for safety. These benchmarks cover a wide range of tasks and dimensions to assess the overall capabilities of large language models. The SuperBench team aims to provide objective and scientific evaluation standards for large models to promote their healthy development in terms of technology, applications, and ecosystem.
SafetyBench

**SafetyBench** is a comprehensive benchmark designed to evaluate the safety of large language models (LLMs) using multiple-choice questions. As LLMs become increasingly prevalent, concerns about their safety have grown. SafetyBench addresses this by providing a reliable evaluation framework for researchers and developers. Here are the key points about SafetyBench:

1. **Purpose**: SafetyBench aims to help researchers and developers better understand and assess the safety of LLMs. It serves as a reference for model selection and optimization, promoting the development of safe, responsible, and ethical large models that align with legislative norms, social standards, and human values¹².

2. **Comprehensive Benchmark**: SafetyBench comprises **11,435 diverse multiple-choice questions** across **7 distinct categories** related to safety concerns. These questions cover a wide range of topics, allowing for thorough evaluation of LLM safety¹.

3. **Multilingual Evaluation**: SafetyBench includes both **Chinese and English data**, making it suitable for evaluating LLMs in both languages. Researchers can assess model safety across different linguistic contexts¹.

4. **Performance Insights**: Extensive tests using SafetyBench on **25 popular Chinese and English LLMs** (including zero-shot and few-shot settings) revealed that **GPT-4** outperformed its counterparts. However, there is still room for improvement in enhancing the safety of existing LLMs¹.

5. **Availability**: Data and evaluation guidelines for SafetyBench are accessible through the following URLs:
   - [SafetyBench Data and Guidelines](https://arxiv.org/abs/2309.07045)
   - [SafetyBench Submission Entrance and Leaderboard](https://arxiv.org/abs/2309.07045)¹⁴.

In summary, SafetyBench provides a valuable resource for assessing and advancing the safety of large language models, contributing to their responsible deployment and alignment with societal norms and values.

(1) [2309.07045] SafetyBench: Evaluating the Safety of Large Language .... https://arxiv.org/abs/2309.07045.
(2) SafetyBench：通过单选题评估大型语言模型安全性. https://posts.careerengine.us/p/6511afb61a8da974e9d62f40.
(3) thu-coai/SafetyBench · Datasets at Hugging Face. https://huggingface.co/datasets/thu-coai/SafetyBench.
(4) SafetyBench：通过单选题评估大型语言模型安全性_鲟曦研习社. https://www.kuxai.com/article/1505.
(5) GitHub - thu-coai/SafetyBench: Official github repo for SafetyBench, a .... https://github.com/thu-coai/SafetyBench.
(6) undefined. https://doi.org/10.48550/arXiv.2309.07045.

Introducing paper: SafetyBench: Evaluating the Safety of Large Language Models
AlignBench

**AlignBench** is a comprehensive benchmark designed specifically for evaluating the alignment performance of large Chinese language models (LLMs). It focuses on assessing how well these models align with human intent across multiple dimensions. Let me provide you with more details:

1. **Purpose and Importance**:
   - For fine-tuned LLMs, alignment with human intent has become a critical factor in their practical applications.
   - Existing evaluation benchmarks do not accurately reflect model performance in real-world scenarios or their alignment with human intent.
   - AlignBench aims to address this challenge by providing a comprehensive, multi-dimensional evaluation benchmark specifically for Chinese LLMs.

2. **Data and Construction**:
   - AlignBench uses a human-in-the-loop data curation process to ensure dynamic and realistic evaluation data.
   - The data comes from real user queries (partly challenging questions constructed by researchers) in ChatGLM online services.
   - It covers various categories, including fundamental language ability, Chinese understanding, open-ended questions, writing ability, logical reasoning, mathematics, task-oriented role play, and professional knowledge.

3. **Evaluation Methodology**:
   - AlignBench employs a multi-dimensional, rule-calibrated evaluation method called "LLM-as-Judge."
   - It combines human judgment with a chain-of-thought analysis to enhance reliability and interpretability.
   - The evaluation process includes comparing model responses to high-quality reference answers and generating multi-dimensional scores.

4. **CritiqueLLM**:
   - To facilitate easy alignment assessment for Chinese researchers, AlignBench introduces a dedicated evaluation model called **CritiqueLLM**.
   - CritiqueLLM can recover 95% of GPT-4's evaluation capability and will provide an accessible API for researchers in the future.

(1) THUDM/AlignBench: 多维度中文对齐评测基准 - GitHub. https://github.com/THUDM/AlignBench.
(2) 智谱AI发布中文 LLM 对齐评测基准AlignBench | 前途科技. https://accesspath.com/ai/5890084/.
(3) AlignBench：量身打造的中文大语言模型对齐评测 - CSDN博客. https://blog.csdn.net/cenyk1230/article/details/135228409.
(4) AlignBench：专为「中文 LLM」而生的对齐评测 - 知乎. https://zhuanlan.zhihu.com/p/671884106.
(5) AlignBench: Benchmarking Chinese Alignment of Large Language Models. https://arxiv.org/abs/2311.18743.

Introducing paper: AlignBench: Benchmarking Chinese Alignment of Large Language Models
AgentBench

**AgentBench** is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) as agents in interactive environments. LLMs, which are increasingly smart and autonomous, have expanded beyond traditional natural language processing tasks to tackle real-world pragmatic missions. Here are the key details about AgentBench:

1. **Purpose**: AgentBench assesses LLMs' reasoning and decision-making abilities in a **multi-turn open-ended generation setting** across challenging tasks.
2. **Environments**: It currently consists of **eight distinct environments**, each representing different scenarios. These environments include:
    - **SQL-based Environment**: Evaluates LLMs' ability to operate with real databases.
    - **Game-based Environment**: Tests LLMs in digital card games.
    - **Web-based Environments**: Includes web shopping, web browsing, and more.
3. **Performance Disparity**: After extensive testing of **27 API-based and open-sourced LLMs**, it was observed that while top commercial LLMs exhibit strong agent capabilities in complex environments, there is a significant performance gap between them and open-source competitors.
4. **Challenges**: Poor long-term reasoning, decision-making, and instruction-following abilities are identified as the main obstacles for developing usable LLM agents.
5. **Improvement Strategies**: Training on code and high-quality multi-turn alignment data could enhance agent performance.
6. **Resources**: Datasets, environments, and an integrated evaluation package for AgentBench are available¹.

In summary, AgentBench provides a rigorous evaluation framework for assessing LLMs' performance as autonomous agents across diverse scenarios. 

(1) [2308.03688] AgentBench: Evaluating LLMs as Agents - arXiv.org. https://arxiv.org/abs/2308.03688.
(2) AGENTBENCH：评估LLMs作为代理的能力 - 知乎 - 知乎专栏. https://zhuanlan.zhihu.com/p/664598024.
(3) AgentBench, a comprehensive benchmark for evaluating AI agent .... https://ai-scholar.tech/en/articles/agent-simulation%2Fagentbench.
(4) AI-Natural-Language-Processing-Lab/AgentBench-LLM-as-Agent. https://github.com/AI-Natural-Language-Processing-Lab/AgentBench-LLM-as-Agent.
(5) AgentBench: Evaluating LLMs as Agents - GitHub. https://github.com/THUDM/AgentBench.
(6) undefined. https://doi.org/10.48550/arXiv.2308.03688.

Introducing paper: AgentBench: Evaluating LLMs as Agents
MathVerse

MathVerse is an innovative benchmark specifically designed to rigorously evaluate the capabilities of Multi-modal Large Language Models (MLLMs) in interpreting and reasoning with visual information in mathematical problems. Developed by a research team from CUHK MMLab and Shanghai Artificial Intelligence Laboratory, MathVerse offers an equitable and comprehensive assessment of MLLMs' ability to understand and process visual diagrams for mathematical reasoning.

The benchmark consists of 2,612 high-quality, multi-subject math problems with diagrams, meticulously collected from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality. This approach results in a total of 15K test samples, allowing MathVerse to thoroughly examine whether and how much MLLMs can genuinely comprehend visual diagrams for solving mathematical problems.

Introducing paper: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
Nvidia's Aegis-AI-Content-Safety-Dataset-1.0

Aegis AI Content Safety Dataset is an open-source content safety dataset (CC-BY-4.0), which adheres to Nvidia's content safety taxonomy, covering 13 critical risk categories (see Dataset Description).

Dataset Details
Dataset Description
The Aegis AI Content Safety Dataset is comprised of approximately 11,000 manually annotated interactions between humans and LLMs, split into 10,798 training samples and 1,199 test samples.

To curate the dataset, we use the Hugging Face version of human preference data about harmlessness from Anthropic HH-RLHF. We extract only the prompts, and elicit responses from Mistral-7B-v0.1. Mistral excels at instruction following and generates high quality responses for the content moderation categories. We use examples in the system prompt to ensure diversity by instructing Mistral to not generate similar responses. Our data comprises four different formats: user prompt only, system prompt with user prompt, single turn user prompt with Mistral response, and multi-turn user prompt with Mistral responses.
Hardpicks dataset

This dataset comprises the raw data of four three-dimensional seismic surveys acquired in Canadian mining camps. They include first-break picks in the trace headers.
Please see all details on the github repo.
SSv2-Spatio-Temporal
Full name: Something Someting v2-Spatio-Temporal

We use Something-Something v2 dataset to obtain the generation prompts and ground truth masks from real action videos. We filter out a set of 295 prompts. The details for this filtering are in the "Peekaboo: Interactive Video Generation via Masked-Diffusion" paper. We then use an off-the-shelf OWL-ViT-large open-vocabulary object detector to obtain the bounding box (bbox) annotations of the object in the videos. This set represents bbox and prompt pairs of real-world videos, serving as a test bed for both the quality and control of methods for generating realistic videos with spatio-temporal control.

Introducing paper: PEEKABOO: Interactive Video Generation via Masked-Diffusion
COCONut

The COCONut dataset is a modernized segmentation dataset that builds upon the established COCO benchmark. It aims to address the limitations of the original COCO segmentation annotations by enhancing annotation quality and expanding the dataset to encompass a larger number of images with high-quality masks. COCONut harmonizes segmentation annotations across semantic, instance, and panoptic segmentation tasks, providing meticulously crafted masks for improved accuracy and consistency. It includes 383K images with more than 5.18M panoptic masks, making it a large-scale universal segmentation dataset verified by human raters [T1], [T6].
Source camera identification

A large image source dataset including 23 devices, 6978 nat images, 2427 flat original images, total 23,361 images, 38G size, with serial numbers and naming conventions standardized. Mimicking the transmission process of social networks, a JPEG compressed dataset transmitted through Ding Talk and WeChat was synchronized and produced for the experiment test work.

Introducing paper: Source Camera Identification with Multi-Scale Feature Fusion Network
Xhate999
Full name: Xhate999

We present XHate-999, a multi-domain and multilingual evaluation data set for abusive language detection. By aligning test instances across six typologically diverse languages, XHate-999 for the first time allows for disentanglement of the domain transfer and language transfer effects in abusive language detection. We conduct a series of domain- and language-transfer experiments with state-of-the-art monolingual and multilingual transformer models, setting strong baseline results and profiling XHate-999 as a comprehensive evaluation resource for abusive language detection. Finally, we show that domain- and language-adaption, via intermediate masked language modeling on abusive corpora in the target language, can lead to substantially improved abusive language detection in the target language in the zero-shot transfer setups.

Introducing paper: XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages