Omkar2118/datasets huggingface dataset:
OpenEQA

The **OpenEQA** dataset is a significant contribution in the field of **Embodied Question Answering (EQA)**. Let me provide you with some details:

1. **Definition**:
   - **Embodied Question Answering (EQA)** involves understanding an environment well enough to answer questions about it in natural language.
   - EQA agents can achieve this understanding through either **episodic memory** (as seen in agents using smart glasses) or **active exploration** of the environment (as in the case of mobile robots).

2. **OpenEQA Dataset**:
   - **OpenEQA** is the **first open-vocabulary benchmark dataset** for EQA that supports both episodic memory and active exploration use cases.
   - It contains over **1600 high-quality human-generated questions** drawn from more than **180 real-world environments**.
   - The dataset consists of **question-answer pairs** ($Q, A^*$) and corresponding **episode histories** ($H$).
   - You can find the question-answer pairs in the file `data/open-eqa-v0.json`.
   - To access the episode histories, follow the instructions provided [here](https://github.com/facebookresearch/open-eqa#dataset).

3. **Evaluation Protocol**:
   - OpenEQA also provides an **automatic evaluation protocol** powered by **language model-based evaluation** (LLM).
   - This evaluation protocol correlates well with human judgment.

4. **Foundation Models Evaluation**:
   - Researchers evaluated several state-of-the-art foundation models, including **GPT-4V**, using the OpenEQA dataset.
   - The findings revealed that these models significantly lag behind **human-level performance** in EQA tasks.

5. **Significance**:
   - OpenEQA serves as a **straightforward, measurable, and practically relevant benchmark** for current-generation foundation models.
   - It poses a considerable challenge and inspires research at the intersection of **Embodied AI**, **conversational agents**, and **world models**.
ScanNet++
Full name: ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

ScanNet++ is a large scale dataset with 450+ 3D indoor scenes containing sub-millimeter resolution laser scans, registered 33-megapixel DSLR images, and commodity RGB-D streams from iPhone. The 3D reconstructions are annotated with long-tail and label-ambiguous semantics to benchmark semantic understanding methods, while the coupled DSLR and iPhone captures enable benchmarking of novel view synthesis methods in high-quality and commodity settings.
HELM
Full name: Holistic Evaluation of Language Models

The Holistic Evaluation of Language Models (HELM) is a comprehensive framework developed by Stanford University for evaluating foundation language models. It serves as a living benchmark, promoting transparency in language models. Here are the key aspects of HELM:

Purpose: HELM aims to provide a holistic view of language models by considering various dimensions and metrics.
Coverage: It encompasses a wide range of scenarios and recognizes the inherent incompleteness of existing models.
Metrics: HELM employs multiple metrics to assess language models.
Standardization: The framework promotes standardization in evaluation practices.
Accessibility: All data and analyses are freely accessible on the HELM website for exploration and study.

Introducing paper: Holistic Evaluation of Language Models
FormNLU
Full name: Form-NLU: Dataset for the Form Language Understanding

We introduce a new dataset for form structure understanding and key information extraction. This repository will provide detailed baseline model descriptions and experimental setups to ensure our model and experiments are reproducible. As well as we will offer a colab link to show how to download and use our dataset for corresponding tasks.

Introducing paper: Form-NLU: Dataset for the Form Natural Language Understanding