Omkar2118/datasets huggingface dataset:
RewardBench

**RewardBench** is a benchmark designed to evaluate the capabilities and safety of reward models, including those trained with **Direct Preference Optimization (DPO)**. It serves as the first evaluation tool for reward models and provides valuable insights into their performance and reliability¹.

Here are the key components of **RewardBench**:

1. **Common Inference Code**: The repository includes common inference code for various reward models, such as **Starling**, **PairRM**, **OpenAssistant**, and more. These models can be evaluated using the provided tools¹.

2. **Dataset and Evaluation**: The **RewardBench dataset** consists of prompt-win-lose trios spanning chat, reasoning, and safety scenarios. It allows benchmarking reward models on challenging, structured, and out-of-distribution queries. The goal is to enhance scientific understanding of reward models and their behavior².

3. **Scripts for Evaluation**:
    - `scripts/run_rm.py`: Used to evaluate individual reward models.
    - `scripts/run_dpo.py`: Used to evaluate direct preference optimization (DPO) models.
    - `scripts/train_rm.py`: A basic reward model training script built on TRL (Transformer Reinforcement Learning)¹.

4. **Installation and Usage**:
    - Install **PyTorch** on your system.
    - Install the required dependencies using `pip install -e .`.
    - Set the environment variable `HF_TOKEN` with your token.
    - To contribute your model to the leaderboard, open an issue on **HuggingFace** with the model name. For local model evaluation, follow the instructions in the repository¹.

Remember that **RewardBench** provides a standardized way to assess reward models, ensuring transparency and comparability across different approaches. 🌟🔍

(1) GitHub - allenai/reward-bench: RewardBench: the first evaluation tool .... https://github.com/allenai/reward-bench.
(2) RewardBench: Evaluating Reward Models for Language Modeling. https://arxiv.org/abs/2403.13787.
(3) RewardBench: Evaluating Reward Models for Language Modeling. https://paperswithcode.com/paper/rewardbench-evaluating-reward-models-for.

Introducing paper: RewardBench: Evaluating Reward Models for Language Modeling
Simple_PS_Dataset

Photometric stereoscopic test data sets under six lights taken using laboratory equipment. Note that this dataset has no GT.
GPQA

**GPQA** stands for **Graduate-Level Google-Proof Q&A Benchmark**. It's a challenging dataset designed to evaluate the capabilities of **Large Language Models (LLMs)** and scalable oversight mechanisms. Let me provide more details about it:

- **Description**: GPQA consists of **448 multiple-choice questions** meticulously crafted by domain experts in **biology, physics, and chemistry**. These questions are intentionally designed to be **high-quality and extremely difficult**.
- **Expert Accuracy**: Even experts who hold or are pursuing **PhDs in the corresponding domains** achieve only **65% accuracy** on these questions (or **74%** when excluding clear mistakes identified in retrospect).
- **Google-Proof**: The questions are "Google-proof," meaning that even with **unrestricted access to the web**, highly skilled non-expert validators only reach an accuracy of **34%** despite spending over 30 minutes searching for answers.
- **AI Systems Difficulty**: State-of-the-art AI systems, including our strongest **GPT-4 based baseline**, achieve only **39% accuracy** on this challenging dataset.

The difficulty of GPQA for both skilled non-experts and cutting-edge AI systems makes it an excellent resource for conducting **realistic scalable oversight experiments**. These experiments aim to explore ways for human experts to reliably obtain truthful information from AI systems that surpass human capabilities¹³.

In summary, GPQA serves as a valuable benchmark for assessing the robustness and limitations of language models, especially when faced with complex and nuanced questions. Its difficulty level encourages research into effective oversight methods, bridging the gap between AI and human expertise.

(1) [2311.12022] GPQA: A Graduate-Level Google-Proof Q&A Benchmark - arXiv.org. https://arxiv.org/abs/2311.12022.
(2) GPQA: A Graduate-Level Google-Proof Q&A Benchmark — Klu. https://klu.ai/glossary/gpqa-eval.
(3) GPA Dataset (Spring 2010 through Spring 2020) - Data Science Discovery. https://discovery.cs.illinois.edu/dataset/gpa/.
(4) GPQA: A Graduate-Level Google-Proof Q&A Benchmark - GitHub. https://github.com/idavidrein/gpqa.
(5) Data Sets - OpenIntro. https://www.openintro.org/data/index.php?data=satgpa.
(6) undefined. https://doi.org/10.48550/arXiv.2311.12022.
(7) undefined. https://arxiv.org/abs/2311.12022%29.

Introducing paper: GPQA: A Graduate-Level Google-Proof Q&A Benchmark
MA-52
Full name: Micro-Action 52 dataset

The MA-52 dataset provides the whole-body perspective including gestures, upper- and lower-limb movements, attempting to reveal comprehensive micro-action cues. In detail, MA-52 contains 52 micro-action categories along with seven body part labels, and encompasses a full array of realistic and natural micro-actions, accounting for 205 participants and 22,422 video instances collated from the psychological interviews

Introducing paper: Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
V2AIX
Full name: A Multi-Modal Real-World Dataset of ETSI ITS V2X Messages in Public Road Traffic

Connectivity is a main driver for the ongoing megatrend of automated mobility: future Cooperative Intelligent Transport Systems (C-ITS) will connect road vehicles, traffic signals, roadside infrastructure, and even vulnerable road users, sharing data and compute for safer, more efficient, and more comfortable mobility. In terms of communication technology for realizing such vehicle-to-everything (V2X) communication, the WLAN-based peer-to-peer approach (IEEE 802.11p, ITS-G5 in Europe) competes with C-V2X based on cellular technologies (4G and beyond). Irrespective of the underlying communication standard, common message interfaces are crucial for a common understanding between vehicles, especially from different manufacturers. Targeting this issue, the European Telecommunications Standards Institute (ETSI) has been standardizing V2X message formats such as the Cooperative Awareness Message (CAM). In this work, we present V2AIX, a multi-modal real-world dataset of ETSI ITS messages gathered in public road traffic, the first of its kind. Collected in measurement drives and with stationary infrastructure, we have recorded more than 230000 V2X messages from more than 1800 vehicles and roadside units in public road traffic. Alongside a first analysis of the dataset, we present a way of integrating ETSI ITS V2X messages into the Robot Operating System (ROS). This enables researchers to not only thoroughly analyze real-world V2X data, but to also study and implement standardized V2X messages in ROS-based automated driving applications.

Introducing paper: V2AIX: A Multi-Modal Real-World Dataset of ETSI ITS V2X Messages in Public Road Traffic
kickstarter
Full name: Funding Successful Projects on Kickstarter

Kickstarter is a community of more than 10 million people comprising of creative, tech enthusiasts who help in bringing creative project to life. Till now, more than $3 billion dollars have been contributed by the members in fueling creative projects. The projects can be literally anything – a device, a game, an app, a film etc.

Kickstarter works on all or nothing basis i.e if a project doesn’t meet it goal, the project owner gets nothing. For example: if a projects’s goal is \$500. Even if it gets funded till \$499, the project won’t be a success.

Recently, Kickstarter released its public data repository to allow researchers and enthusiasts like us to help them solve a problem. Will a project get fully funded ?