PRELUDE: A Benchmark Designed to Require Global Comprehension and
  Reasoning over Long Contexts
Authors: name: Mo Yu; name: Tsz Ting Chung; name: Chulun Zhou; name: Tong Li; name: Rui Lu; name: Jiangnan Li; name: Liyan Xu; name: Haoshu Lu; name: Ning Zhang; name: Jing Li; name: Jie Zhou
Published: 2025-08-13 14:28:25

Abstract: We introduce PRELUDE, a benchmark for evaluating long-context understanding
through the task of determining whether a character's prequel story is
consistent with the canonical narrative of the original book. Our task poses a
stronger demand for global comprehension and deep reasoning than existing
benchmarks -- as the prequels are not part of the original story, assessing
their plausibility typically requires searching and integrating information
that is only indirectly related. Empirically, 88% of instances require evidence
from multiple parts of the narrative. Experimental results highlight the
challenge of our task: in-context learning, RAG and in-domain training with
state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans
by >15%. A further human study reveals that models often produce correct
answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy
compared to humans. These findings underscore the substantial room for
improvement in long-context understanding and reasoning.