Title: One Thousand and One Pairs: A “novel” challenge for long-context language models
Benchmark: NOCHA
Summary: While synthetic long-context LLM benchmarks typically test surface-level retrieval, the NOCHA dataset assesses models' abilities to retrieve, synthesize, and reason over book-length texts. The dataset consists of true and false claim pairs about 67 recently-published English fictional books, requiring global reasoning for verification. Experiments show that human readers excel at this task, but long-context LLMs struggle significantly, with the highest accuracy from GPT-4O at 55.8%, indicating a need for improved models and methodologies for handling extensive world-building and complex narratives.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Claim verification in long-context tasks
Phenomenon defined by authors: Yes
Phenomenon definition: true/false narrative minimal pairs that isolate a single narrative phenomenon present in their novels. Each false claim differs from the true claim in its pair only by the inclusion of false information regarding the same event or entity
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Discern true and false claims about books
Task item definition: Book, claim, answer, score
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1001
Dataset metadata available: Yes
Dataset metadata detail: global, passage, sentence
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum, GLMs
Metric subscores: Yes
Metric subscores detail: global, passage, sentence
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: realistic dataset
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context