Title: SportQA: A Benchmark for Sports Understanding in Large Language Models
Benchmark: SportQA
Summary: We introduce SportQA, a novel benchmark specifically designed for evaluating LLMs in the context of sports understanding. We conducted a thorough evaluation of prevalent LLMs, mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Sports Understanding
Phenomenon defined by authors: Yes
Phenomenon definition: A deep understanding of sports, a field rich in strategic and dynamic content, is crucial for advancing Natural Language Processing (NLP).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Take a question on sports understanding + multiple choice, provide a choice. 
Task item definition: Input: a question on sports understanding + multiple choice, output:  provide a choice. 
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: Yes
Dataset metadata detail: Type of sports
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: Type of sports. 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Sports / 
New taxonomy: Specific Application / Profesisonal Domains