Title: LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
Benchmark: Arena-Hard-200
Summary: The paper introduces LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset was collected from 210K unique IP addresses through the Vicuna demo and Chatbot Arena website. The authors demonstrate the dataset's versatility through four use cases: developing content moderation models, building safety benchmarks, training instruction-following models, and creating challenging benchmark questions (this is the Arena-Hard Benchmark).
Contribution: It introduces the first large-scale real-world LLM conversation dataset (LMSYS-Chat-1M) with 1 million user conversations with different LLMs
It provides analysis and visualisation of the distribution of user queries
It demonstrates multiple practical applications including content moderation, safety benchmarking, instruction-following model training, and creating challenging benchmark questions (Arena-Hard-200)
The dataset contains conversations from 25 different LLMs, offering a diverse range of model responses and user interactions in open and closed source models.
The paper attempt ecological validity by capturing real-world interactions rather than synthetic data
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLM capabilities in real-world user interactions, including problem-solving, creativity, and adherence to real-world facts. Particularly Arena-Hard-200 focuses on "challenging" prompts.
Phenomenon defined by authors: Yes
Phenomenon definition: For challenging: they say - "we consider  a prompt to be challenging if it requires integrating various knowledge and skills to derive appropriate responses." but they do note "Defining what constitutes 'challenging' prompts is essential in crafting benchmark questions. While there are many definitions that could address topics ranging from ethical and philosophical reasoning to problem-solving and information retrieval."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: The authors note the difficulty in benchmarking LLMs as their skills have grown more advanced and recognize that real-world tasks require integration of diverse skills such as problem-solving, creativity, knowledge, and common sense. The Arena-Hard benchmark specifically focuses on challenging prompts that require integrating multiple skills, while acknowledging this is just one definition of "challenging" among many possible interpretations. They also focus only on "good" prompts and provide specific examples of what constitutes "good prompts" for their benchmark, such as prompts that require explaining complex concepts in simple terms (e.g., "Can you explain gravity to a 10-year-old with a simple example"), prompts that require comparative analysis of fictional languages, and prompts that test mathematical problem-solving abilities. In contrast, they identify "bad prompts" as those that are too straightforward or narrow (e.g., "How is it going today?" or "What is my IP address?").
Task definition: The task is to very broad - to evaluate LLMs on challenging, real-world prompts from users that test diverse skills such as problem-solving, creativity, knowledge integration, and adherence to facts.
Task item definition: A single item consists of a challenging user prompt from the LMSYS-Chat-1M dataset e.g., "Implement FizzBuzz in a short perl script and annotate it in the style of Shakespeare."
Task definition detail: The authors curated Arena-Hard-200, consisting of the 200 most challenging and high-quality user prompts extracted from the Chatbot Arena subset of LMSYS-Chat-1M. These prompts were selected based on scoring by multiple LLMs (GPT-3.5-Turbo, Claude-2, and GPT-4) and required a score of 9+ to be included.
Task source: Human-sourced task examples (not crowdworkers per say as these are non-paid real-users)
Task source detail: The prompts are taken from real user interactions with LLMs on the Chatbot Arena website. The dataset was collected from 210K unique IP addresses from April to August 2023, containing conversations with 25 different state-of-the-art LLMs.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The dataset has strong ecological validity as it contains real-world interactions between users and LLMs. The authors specifically note that "studying how people interact with LLMs in real-world scenarios is increasingly important" and emphasize the gap their dataset fills by providing "diverse," "original," and "real-world" conversations.
Dataset size: 200
Dataset size detail: Arena-Hard-200 consists of 200 most challenging prompts selected from a larger set of real-world conversations based on specific scoring criteria.
Dataset size notes: LMSYS-1M is also released as a training dataset. - Unclear if Arena-Hard-200 are actually removed from this wider dataset, if not there could be leakage.
Dataset metadata available: No
Dataset metadata detail: Note in the analysis of LMSYS they provide a lot of detail e.g., topic, language of queries etc but not for Arena-Hard-200
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean, A bit unclear what they are actually showing in Fig 1 - I think it must be an average score across 200 prompts but it just says Score (0-10) on x-axis label
Metric subscores: No
Metric subscores detail: Note all of the set is a challenging test set. But no "easy" test set is provided.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: The link to LMSYS is provided: https://huggingface.co/datasets/lmsys/lmsys-chat-1m but Arena-Hard-200 doesn't seem to be avaliable?
Task face validity: Maybe - good on ecological validity but a very small and specific set of 200 prompts
Metric face validity: Maybe: You could imagine that GPT-4 is of lower capability than the model being evaluated which would mean it couldn't necessarily judge what a good or correct answer is.
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors provide evidence for the validity of their benchmark through an ablation study. They designed a test where they compared responses of GPT-4 against GPT-3.5-Turbo on two subsets of prompts: high-scoring (>8) and low-scoring (<2). They found that "GPT-4 wins 52% in Top-50 but only 22% in Bottom-50 against GPT-3.5-turbo, suggesting Top-50 prompt set is much more effective in benchmarking models." This demonstrates that their scoring and selection approach effectively identifies prompts that can distinguish between model capabilities.
Additionally, they compare Arena-Hard-200 to MT-Bench and observe that Arena-Hard-200 "reveals larger performance gaps between open and proprietary models (e.g., GPT-4, Claude) than MT-Bench, suggesting more rooms for open models to catch up in this challenging real-world task set."
Taxonomy: General Purpose / 
New taxonomy: General Capability / User Interaction
Other notes: Basically this paper is mainly releasing the dataset then introduces the Arena-Hard-200 benchmark as an EXAMPLE of how it can be used. They also say they create a safety benchmark of demonstrated jailbreaks but the details are very scant (the benchmark has no name as far as I can tell). So it's possible that Arena-Hard-200 is not intended as a standalone benchmark for others to use but to serve as a demonstration of how the wider dataset could be used??

One of my bigger concerns is whether these constructed benchmarks are removed /held-out from LMSYS dataset itself, or if there could be contamination if others later report on the Arena-Hard-200 benchmark and also train on LMSYS.