Title: UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis
Benchmark: UDA
Summary: The paper introduces the UDA (Unstructured Document Analysis) benchmark. UDA questions are expert-annotated Q&A pairs on PDF and HTML documents, constructed from datasets of academic papers, financial reports, and Wikipedia pages.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Analysing unstructured documents
Phenomenon defined by authors: No
Phenomenon definition: Vague and multifaceted: "we propose a benchmark suite that enables the evaluation of various components of RAG-based unstructured document analysis"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: LLMs are given an unstructured document and a factual question about the contents of that document. The correct answer is some extracted text or figure from the document.
Task item definition: An unstructured document might be a financial report in PDF format, containing tabular data. The question might ask for the total value of some column, like "total vested shares during the 2012 fiscal year, in millions," and correct answers might be [1.46, 1.45972].
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Hand-written answers are "expert annotated" by the authors of six Q&A datasets; the authors curate and filter these without changing the labels.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 29,590
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: "Free responses" are intended to be extracted from the provided file's text.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Simple mean/sum; % improvement between contexts
Metric subscores: Yes
Metric subscores detail: Scores by underlying Q&A dataset, context type (whether document chunks are provided by RAG or by human annotators)
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval