Omkar2118/datasets huggingface dataset:
LMC
Full name: Language Model Council

The **Language Model Council (LMC)** is a novel benchmarking framework proposed to address the challenge of ranking Large Language Models (LLMs) on highly subjective tasks¹. These tasks can include areas related to emotional intelligence, creative writing, or persuasiveness, which often lack majoritarian human agreement¹.

The LMC operates through a democratic process to¹:
1. Formulate a test set through equal participation,
2. Administer the test among council members, and
3. Evaluate responses as a collective jury¹.

The council is composed of the newest LLMs and they are deployed on an open-ended emotional intelligence task: responding to interpersonal dilemmas¹. The results from the LMC have shown to produce rankings that are more separable, robust, and less biased than those from any individual LLM judge¹. It is also more consistent with a human-established leaderboard compared to other benchmarks¹.

This benchmarking framework is designed to encourage the healthy development of the field, particularly through the lens of mathematical reasoning tasks². It provides a more comprehensive and fair comparison of different LLMs, especially for tasks that are highly subjective and often lack majoritarian human agreement¹.

(1) Language Model Council: Benchmarking Foundation Models on Highly .... https://arxiv.org/abs/2406.08598.
(2) GitHub - GAIR-NLP/benbench: Benchmarking Benchmark Leakage in Large .... https://github.com/GAIR-NLP/benbench.
(3) LLM Benchmarks: Understanding Language Model Performance. https://humanloop.com/blog/llm-benchmarks.
(4) One Billion Word Benchmark for Measuring Progress in Statistical .... https://research.google.com/pubs/pub41880.html?source=post_page---------------------------.

Introducing paper: Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks