Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
  Multi-Doc QA
Authors: name: Minzheng Wang; name: Longze Chen; name: Cheng Fu; name: Shengyi Liao; name: Xinghua Zhang; name: Bingli Wu; name: Haiyang Yu; name: Nan Xu; name: Lei Zhang; name: Run Luo; name: Yunshui Li; name: Min Yang; name: Fei Huang; name: Yongbin Li
Published: 2024-06-25 09:42:56

Abstract: Long-context modeling capabilities have garnered widespread attention,
leading to the emergence of Large Language Models (LLMs) with ultra-context
windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
catching up. However, existing benchmarks employ irrelevant noise texts to
artificially extend the length of test cases, diverging from the real-world
scenarios of long-context applications. To bridge this gap, we propose a novel
long-context benchmark, Loong, aligning with realistic scenarios through
extended multi-document question answering (QA). Unlike typical document QA, in
Loong's test cases, each document is relevant to the final answer, ignoring any
document will lead to the failure of the answer. Furthermore, Loong introduces
four types of tasks with a range of context lengths: Spotlight Locating,
Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
and comprehensive evaluation of long-context understanding. Extensive
experiments indicate that existing long-context language models still exhibit
considerable potential for enhancement. Retrieval augmented generation (RAG)
achieves poor performance, demonstrating that Loong can reliably assess the
model's long-context modeling capabilities.