Title: Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models
Benchmark:  Shopping MMLU
Summary:  Shopping MMLU is a comprehensive benchmark for evaluating how large language models (LLMs) perform on online shopping tasks.  The authors transformed online shopping tasks into a text-to-text format suitable for LLMs, evaluated over 20 different models, and analyzed performance patterns.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: performance on online shopping tasks
Phenomenon definition: "Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Models are presented with shopping-related prompts (such as product descriptions, user queries, or reviews) and must generate appropriate responses (like classifications, rankings, entity extraction, or product recommendations) following specific instructions. 
Task item definition: shopping-related prompts (such as product descriptions, user queries, or reviews) 

Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: It includes real-world shopping elements (actual product descriptions, genuine user queries, etc) but the tasks are presented as isolated questions rather than as part of a complete interactive shopping experience
Dataset size:  57 tasks and 20,799 questions
Dataset metadata available: No
Dataset metadata detail: Task type: shopping concept understanding, shopping knowledge reasoning, user behaviour alignment, multi-lingual abilities
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple average 
Metric subscores: Yes
Metric subscores detail: For task type
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail:  The authors validate their benchmark by comparing LLM performance against task-specific state-of-the-art methods on three representative tasks.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents