Title: QTSumm: Query-Focused Summarization over Tabular Data
Benchmark: QTSUMM
Summary: Propose a dataset. Show that the task is challenging. Propose a method to improve model performance.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: table summarization
Phenomenon defined by authors: Yes
Phenomenon definition: "the model is required to generate a user-customized summary given the table and user query"-p1158
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: "the model is required to generate a user-customized summary given the table and user query"-p1158
Task item definition: given a query and a table, the model has to generate a user-customized summary
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 440 tables, 1078 summaries
Dataset size notes: train: 2055 tables, 4981 summaries; dev: 439 tables, 1052 summaries
Dataset metadata available: Yes
Dataset metadata detail: topic area, length, and annotation details
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric subscores: No
Metric access: Model access required (e.g. logits)
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: they conduct extensive human experiment
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis