Title: SyllabusQA: A Course Logistics Question Answering Dataset
Benchmark: SyllabusQA
Summary: The paper introduces a new dataset consisting of real-world syllabi for question-answering. Strong LLMs are benchmarked on the dataset, SyllabusQA.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: course logistics-related question-answering
Phenomenon defined by authors: Yes
Phenomenon definition: "answering student questions on logistics whose answers can be directly found or inferred from the syllabus"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take a syllabus and question and respond using information from the syllabus.
Task item definition: Each sample is represented with the syllabus name, question type, question, and answer, along with meta-data indicating the sample index, answer spans (if applicable), and reasoning steps (if applicable).
Task source: Real task examples (e.g. GitHub issues), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1103
Dataset size notes: Train: 3018; Validation: 957
Dataset metadata available: Yes
Dataset metadata detail: pages per syllabus, tokens per syllabus, tokens per question, tokens per answer
Sampling method: Unknown
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: question type, answer source type
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval