Title: FanOutQA: A Multi-Hop, Multi-Document  Question Answering Benchmark for Large Language Models
Benchmark: FanOutQA
Summary: Benchmark to test LLM performance on "Fan-Out" questions, that require models to acquire information from multiple sources and combine. Test on 3 settings - closed-book (no retrieval), open-book (answer with retrieval / search) and evidence-provided (given answers to sub questions combine them).
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: The ability to answer “fan-out” questions: questions that require models to find a list of entities and then consult a large number of documents to aggregate information about those entities to answer a user’s question.
Phenomenon defined by authors: Yes
Phenomenon definition: “fan-out” questions: questions that require models to find a list of entities and then consult a large number of documents to aggregate information about those entities to answer a user’s question.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: We formulate three distinct challenge settings over the dataset. The closed-book setting requires  the model to answer fan-out questions without external knowledge, testing its general knowledge. The open-book setting gives models access to retrieval tools, testing their ability to retrieve relevant articles and reason across multiple long documents. Finally, the evidence-provided setting provides the models with relevant articles, testing their long context and multi-hop reasoning capabilities.
Task item definition: Q: What is the total number of employees in the five largest banks in the world?
A: 1,604,898

Additional metainformation:
Suggested sub-questions, relevant documents, and answers, ie Q:"How many employees does Bank of America have? Document: ...... A: 217,000
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 724
Dataset size notes: 310 validation questions
Dataset metadata available: Yes
Dataset metadata detail: Each question has human written decomposition into sub questions. Each sub question is attached to its answer and the original document that provided the answer.
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Task has 3 difficulty levels: "Open book", "closed book" and "evidence provided".
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval