Title: A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models
Summary: The paper creates a dataset of user scenarios for LLMs based on actual survey data, then collects reponses and rates them with GPT 4, before validating those ratings with human preferences.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: User reported scenarios
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Respond to user-generated questions
Task item definition: A single question prompt from the user survey
Task source: Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1024
Dataset metadata available: Yes
Dataset metadata detail: type of task, language of task, country of task author
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: task type and country
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: It seems unlikely that so broad a concept could be measured well, but this is a good effort to cast a wide net.
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They use a human validation to compare to the LLM judge
Taxonomy: General Purpose / 
New taxonomy: General Capability / User Interaction