In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss
Authors: name: Yuri Kuratov; name: Aydar Bulatov; name: Petr Anokhin; name: Dmitry Sorokin; name: Artyom Sorokin; name: Mikhail Burtsev
Published: 2024-02-16 16:15:01

Abstract: This paper addresses the challenge of processing long documents using
generative transformer models. To evaluate different approaches, we introduce
BABILong, a new benchmark designed to assess model capabilities in extracting
and processing distributed facts within extensive texts. Our evaluation, which
includes benchmarks for GPT-4 and RAG, reveals that common methods are
effective only for sequences up to 10^4 elements. In contrast, fine-tuning
GPT-2 with recurrent memory augmentations enables it to handle tasks involving
up to 10^7 elements. This achievement marks a substantial leap, as it is by
far the longest input processed by any open neural network model to date,
demonstrating a significant improvement in the processing capabilities for long
sequences.