In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss
Authors: name: Yuri Kuratov, name: Aydar Bulatov, name: Petr Anokhin, name: Dmitry Sorokin, name: Artyom Sorokin, name: Mikhail Burtsev

This paper addresses the challenge of processing long documents using
generative transformer models. To evaluate different approaches, we introduce
BABILong, a new benchmark designed to assess model capabilities in extracting
and processing distributed facts within extensive texts. Our evaluation, which
includes benchmarks for GPT-4 and RAG, reveals that common methods are
effective only for sequences up to 10^4 elements. In contrast, fine-tuning
GPT-2 with recurrent memory augmentations enables it to handle tasks involving
up to 10^7 elements. This achievement marks a substantial leap, as it is by
far the longest input processed by any open neural network model to date,
demonstrating a significant improvement in the processing capabilities for long
sequences.