Title: DataTales: A Benchmark for Real-World Intelligent Data Narration
Benchmark: DataTales
Summary: DataTales is a novel benchmark designed to assess data narration of market movement data. It contains a human baseline and is publicly available. Specifically, DataTales assesses the proficiency of LLMs at performing lookups, comparisons, subtraction, rate of change, causal analysis, trend analysis, and predictive analysis to craft a financial report based upon market data. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data narration
Phenomenon defined by authors: Yes
Phenomenon definition: Data narration is the process of transforming intricate data into compelling narratives.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: DataTales focuses on data narration with financial data, e.g. narrating financial market reports. Data Narration is separated into seven analytical operations across three domains: simple lookup, basic quantitative operations, and advanced analytical operations. Basic quantitative operations include comparison, subtraction, and rate of change, while advanced analytical operations include causal analysis, trend analysis, and predictive analysis. 
Task definition: We define the task of financial data narration as follows: given market movement data  {T_{i,j} | i ≤ E_T, j ≤ D_T } with E_T financial entities and D_T days, where T_{i, j} is the row of entity i on date j, a data narration model M generates a report y narrating the market data y = M(T_{i,j} | i ≤ E_T, j ≤ D_T). Narrations are evaluated generated with same-day data, and historical data spanning one week. Both zero-shot and fine-tuned scenarios are analyzed. 
Task item definition: A single item in the dataset would have market movement data (open, high, low, close, volume), the date, the entity, the market report, and the market type. 
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 4900
Dataset size notes: Train/Validation and Testing 80/20, split by time
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Accuracy is calculated with a specific MCQA-inspired methodology, that utilizes Named Entity Recognition to assess if LLM's predict numerical values accurately. 
Metric aggregation: Simple Mean, None
Metric statistics: Factuality is calculated with Named Entity Recognition (NER) empowered accuracy, described in the paper. Style is measured with BLEU. Insightfulness is measured by human assessments based on impact (breadth of claim), and significance (magnitude of changes) on a 5 point Likert scale, and the average of the human review is reported. 
Metric subscores: Yes
Metric subscores detail: Factuality, Style, and Insightfulness
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors highlight that data narration requires deeper analysis to craft narratives around key insights, and goes beyond the scope of existing datasets that focus on data-to-text tasks like basic information transformation. Thus, this justification is used to define a benchmark exclusively tailored for data narration. 
Taxonomy: Data Analysis / 
New taxonomy: General Application / Data Analysis