Title: SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs
Benchmark: SportsMetrics
Summary: SportsMetrics evaluates LLMs' numerical reasoning abilities within a sports domain. Specifically, it tasks LLMs with filling in information based on play-by-play descriptions from different games. SportsMetrics also include adversarial examples with scrambled rules. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Numerical reasoning and (numerical) information fusion
Phenomenon defined by authors: Yes
Phenomenon definition: "Information fusion focuses on synthesizing information from multiple textual sources to derive meaningful conclusions" (numerical reasoning is more vaguely defined as ability to "tackle mathematical word problems")
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: The authors narrow down the scope by focusing specifically on the domain of sports. 
Task definition: The task is generally to keep track of either the points or comprehensive game statistics given a partial play-by-play description of the game.
Task item definition: Each task has a game recap (play-by-play) and a description of target statistics (e.g., the final score, and the rebounds for a specific player) in cloze style.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Reconstructing summary statistics from a game is likely to be relatively automatable without LLMs. Still, the general idea of extracting numerical data from long texts is fairly realistic.
Dataset size: 200
Dataset size notes: 34359
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: There is some flexibility in the exact internal organisation of the data structure, but it has to be JSON
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Specfically, the metric is delta $TARGET, where target can be, e.g., the ground truth point score. Note, there is no discussion of how this relates to information fusion. 
Metric aggregation: Simple Mean
Metric statistics: Simple summary stats. 
Metric subscores: Yes
Metric subscores detail: For both individual and aggregated metrics.
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: No
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning