Omkar2118/datasets huggingface dataset:
Vibe-Eval

Vibe-Eval is a new open benchmark and framework for evaluating multimodal chat models¹². It was introduced by Reka Technologies⁴ and is designed to rigorously test these models' visual understanding capabilities⁴. Here are some key points about Vibe-Eval:

- It consists of **269 ultra high-quality image-text prompts** and their ground truth responses¹.
- The prompts and responses have been extensively checked multiple times by the Reka team¹.
- Vibe-Eval is designed to be difficult, challenging even to the current frontier models, and to induce greater separability among frontier-class models¹.
- On 50% of the hard set, all frontier models fail to arrive at a perfect answer, leaving a lot of headroom for progress¹.
- The prompts are created by actual AI experts who have a strong familiarity with the performance of frontier models¹.
- While MMMU has been a pretty solid standard for evaluating multimodal models, it is still fundamentally a multiple-choice benchmark¹. Vibe-Eval, on the other hand, is an open-ended evaluation setup¹.
- They also discuss challenges and trade-offs between human and model-based automatic evaluation and propose a lightweight automatic evaluation protocol based on Reka Core¹.
- They plan to periodically run formal human evaluations on public models that do well on this benchmark¹.

(1) Vibe-Eval: A new open and hard evaluation suite for measuring progress .... https://www.reka.ai/news/vibe-eval.
(2) Vibe-Eval: A hard evaluation suite for measuring progress of multimodal .... https://arxiv.org/pdf/2405.02287.
(3) This AI Paper by Reka AI Introduces Vibe-Eval: A Comprehensive Suite .... https://www.marktechpost.com/2024/05/02/this-ai-paper-by-reka-ai-introduces-vibe-eval-a-comprehensive-suite-for-evaluating-ai-multimodal-models/.
(4) Vibe-Eval: A hard evaluation suite for measuring progress of multimodal .... https://arxivtools.blob.core.windows.net/xueshuxiangzipaperhtml/2024_5_6/2405.02287.pdf.

Introducing paper: Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
NIAN
Full name: Needle in a Needlestack

The **Needle in a Needlestack** (NIAN) is a new benchmark designed to measure how well Language Learning Models (LLMs) pay attention to the information in their context window¹. 

In this benchmark, a prompt is created that includes thousands of limericks, and the prompt asks a question about one limerick at a specific location¹. The goal is to test the ability of LLMs to locate and understand specific information within a large context. 

For example, a prompt might include around 2500 limericks, and the LLM is asked a question about a limerick at a certain position¹. The LLM's task is to correctly answer the question by paying attention to the right part of the context.

(1) GPT-4o’s Memory Breakthrough! (NIAN code) | needle-in-a-needlestack. http://nian.llmonpy.ai/.
(2) [P] Needle in a Needlestack (NIAN) | allainews.com. https://allainews.com/item/p-needle-in-a-needlestack-nian-2024-05-16/.
(3) GitHub - llmonpy/needle-in-a-needlestack. https://github.com/llmonpy/needle-in-a-needlestack/.
CatFLW

The Cat Facial Landmarks in the Wild (CatFLW) dataset contains 2079 images of cats' faces in various environments and conditions, annotated with 48 facial landmarks and a bounding box on the cat’s face.

Introducing paper: Automated Detection of Cat Facial Landmarks
Leipzig Corpora

The Leipzig Corpora Collection presents corpora in different languages using the same format and comparable sources. All data are available as plain text files and can be imported into a MySQL database by using the provided import script. They are intended both for scientific use by corpus linguists as well as for applications such as knowledge extraction programs.
The corpora are identical in format and similar in size and content. They contain randomly selected sentences in the language of the corpus and are available in sizes from 10,000 sentences up to 1 million sentences. The sources are either newspaper texts or texts randomly collected from the web. The texts are split into sentences. Non-sentences and foreign language material was removed. Because word co-occurrence information is useful for many applications, these data are precomputed and included as well. For each word, the most significant words appearing as immediate left or right neighbor or appearing anywhere within the same sentence are given. More information about the format and content of these files can be found here.
The corpora are automatically collected from carefully selected public sources without considering in detail the content of the contained text. No responsibility is taken for the content of the data. In particular, the views and opinions expressed in specific parts of the data remain exclusively with the authors.

If you use one of these corpora in your work we kindly ask you to cite this paper as

D. Goldhahn, T. Eckart & U. Quasthoff: Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.
In: Proceedings of the 8th International Language Resources and Evaluation (LREC'12), 2012