Title: MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations
Benchmark: MMLongBench-Doc
Summary: The paper presents a long-context multimodal benchmark dataset of more than 1k expert annotated questions over long PDFs which require aggregating evidence across multiple locations and evidence formats (text, image, charts, etc.) to answer. MMLongBench-Doc presents a challenge for strong models such as GPT-4o and other large vision language models (LVLMs), demonstrating the need for improved long-context LVLM capabilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context document understanding
Phenomenon defined by authors: Yes
Phenomenon definition: "the automatic understanding of [long-context] documents. The understanding of these lengthy documents brings new challenges for LVLMs", including localization and cross-page comprehension 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Give a document to a model and have it answer a question regarding information in the document.
Task item definition: Documents are PDF files. Questions are stored in json format with the following attributes: document ID, document type, question, answer, evidence pages, evidence sources, and answer format.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1082
Dataset metadata available: Yes
Dataset metadata detail: evidence source, answer format, question length statistics, answer length statistics, document length statistics
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: type of evidence source, number of evidence pages involved in answering the question, document type
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context