Title: Needle In A Multimodal Haystack
Benchmark: Needle In A Multimodal Haystack (MM-NIAH)
Summary: MM-NIAH is a multimodal (image+text) variant of the conventional "needle in a haystack" task in NLP. Authors concatenate documents from OBELICS to produce long-context documents, embed sentences into text or artifacts into images, and prompt MLLMs to answer questions about these "needles."
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context multimodal understanding
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: "For a comprehensive evaluation, we design three types of tasks, including retrieval, counting, and reasoning in our MM-NIAH" (2). Retrieval is standard multimodal IR, counting refers to counting artefacts in the document, and reasoning can be visual-compositional, commonsense, etc.
Task item definition: Authors might take an arbitrary long-context multimodal document and introduce textual needles like "The penguin counted 2 bananas," then ask at the end of the context "How many bananas did the penguin count in total?" Equivalently, small images will be overlayed on images in the document and asked about at the end.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 18,000
Dataset size detail: task_dataset_size is for train; task_dataset_size_extra is for validation; test split is stated to exist in the paper but there is no record of it or a stated size
Dataset size notes: 520
Dataset metadata available: Yes
Dataset metadata detail: needle placement depth (in % of context), total context length, needle modality (image or text), task type
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Soft Accuracy for counting task
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: heatmap of average accuracy by context length and position of needle in document
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: VQA / Long Context
New taxonomy: General Capability / Long Context