LongGenBench: Long-context Generation Benchmark
Authors: name: Xiang Liu; name: Peijie Dong; name: Xuming Hu; name: Xiaowen Chu
Published: 2024-10-05 15:33:25

Abstract: Current long-context benchmarks primarily focus on retrieval-based tests,
requiring Large Language Models (LLMs) to locate specific information within
extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
Long-context generation refers to the ability of a language model to generate
coherent and contextually accurate text that spans across lengthy passages or
documents. While recent studies show strong performance on NIAH and other
retrieval-based long-context benchmarks, there is a significant lack of
benchmarks for evaluating long-context generation capabilities. To bridge this
gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
LongGenBench, which allows for flexible configurations of customized generation
context lengths. LongGenBench advances beyond traditional benchmarks by
redesigning the format of questions and necessitating that LLMs respond with a
single, cohesive long-context answer. Upon extensive evaluation using
LongGenBench, we observe that: (1) both API accessed and open source models
exhibit performance degradation in long-context generation scenarios, ranging
from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
performance degradation, with the Gemini-1.5-Flash model showing the least
degradation among API accessed models, and the Qwen2 series exhibiting the
least degradation in LongGenBench among open source models.