Is It Really Long Context if All You Need Is Retrieval? Towards
  Genuinely Difficult Long Context NLP
Authors: name: Omer Goldman; name: Alon Jacovi; name: Aviv Slobodkin; name: Aviya Maimon; name: Ido Dagan; name: Reut Tsarfaty
Published: 2024-06-29 11:09:47

Abstract: Improvements in language models' capabilities have pushed their applications
towards longer contexts, making long-context evaluation and development an
active research area. However, many disparate use-cases are grouped together
under the umbrella term of "long-context", defined simply by the total length
of the model's input, including - for example - Needle-in-a-Haystack tasks,
book summarization, and information aggregation. Given their varied difficulty,
in this position paper we argue that conflating different tasks by their
context length is unproductive. As a community, we require a more precise
vocabulary to understand what makes long-context tasks similar or different. We
propose to unpack the taxonomy of long-context based on the properties that
make them more difficult with longer contexts. We propose two orthogonal axes
of difficulty: (I) Diffusion: How hard is it to find the necessary information
in the context? (II) Scope: How much necessary information is there to find? We
survey the literature on long-context, provide justification for this taxonomy
as an informative descriptor, and situate the literature with respect to it. We
conclude that the most difficult and interesting settings, whose necessary
information is very long and highly diffused within the input, is severely
under-explored. By using a descriptive vocabulary and discussing the relevant
properties of difficulty in long-context, we can implement more informed
research in this area. We call for a careful design of tasks and benchmarks
with distinctly long context, taking into account the characteristics that make
it qualitatively different from shorter context.