InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Authors: Chaojun Xiao
Pengle Zhang
Xu Han
Guangxuan Xiao
Yankai Lin
Zhengyan Zhang
Zhiyuan Liu
Maosong Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Long context
Large Language Models
Primary area: natural_language_processing
TL;DR: We introduce a training-free memory-based method, InfLLM, to unveil the intrinsic ability of LLMs to process streaming long sequences.

Abstract: Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to 1,024K, InfLLM still effectively captures long-distance dependencies. Our code can be found at https://github.com/thunlp/InfLLM.

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Chaojun Xiao' " Pengle Zhang' Xu Han Guangxuan Xiao Yankai Lin", Zhengyan Zhang', Zhiyuan Liu't, Maosong Sun
!NLP Group, DCST, IAI, BNRIST, Tsinghua University =Quan Cheng Laboratory SShangaii Artificial Intelligence Laboratory Massachusetts Institution of Technology 'Renmin University of China xiaoc 200aac.00mmals.ttsaa edu. en, (hanxu2022, liuzy) (eteettinnngi edu. cn
Abstract
Large language models (LLMs) have emerged as comnerstone in real-world ap- plications with lengthy streaming inputs (c.g., LLM-driven agents). However, existing LLMs, pre-trained sequences with restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training introduce expensive computational overhead and uncontrollable change in model capabillities. In this paper. unveil the intrinsic capacity of LLMs for understand- ing extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfL.LM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to 1, 024K, InfLLLM still effectively captures long-distance dependencies. Our code can be found at http:///gitub..om/ttun////t
longer sequences, which will
Introduction
Recently, large language models (LL.Ms) have achieved profound accomplishments in various tasks (Brown et al., 2020; Bommasani al., 2021; Han et al., 2021; Touvron et al., 2023; Meta, 2024). Their ability to follow complex instructions shed light on the realization of artificial general intelligence (OpenAI, 2023; Ouyang et al., 2022). With the blooming of LLM-driven applications, such as agent construction (Park et al., 2023; Qian et al., 2023; Wang et al., 2024a) and embodied roboties (Driess et al., 2023; Liang al., 2023), enhancing the capability of LLMs to process streaming long sequences become increasingly crucial. For instance, LLM L-M-dríven agents are required process information continuously received from external environments based on all their historical memories, necessitating a robust capability for handling long streaming sequences.
Due to limitations caused by unseen lengthy inputs (Han al., 2023) and distracting noisy con- texts (Liu et al., 2023; Tworkowski et al., 2023), most LLMs, pre-trained on sequences consisting of only few thousand tokens, cannot process longer sequences (Press al., 2022; Zhao et al., 2023).
"Equal contribution. CCorreppondigg authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Common solutions usually involve continually training LLMs on longer sequences but further result in substantial costs and require large-scale high-quality long-sequence datasets (Xiong et al., 2023; Li et al., 2023). And the continual training process on longer sequences may weaken the performance of L.MMs on short contexts (Ding et al., 2024). view of this, improving the length generalizability of LLMs without further training receives extensive attention, trying to make LLMs trained on short sequences directly applicable to long sequences.
In this paper, we propose a training-free memmorybaseed approach, named InfLLM, for streamingly processing extremely long sequences with limited computational costs. Specifically, InfLLM incor- porate the sliding window attention (Xiao et al., 2023; Han et al., 2023) with an efficient context memory, where each token only attends to local contexts and relevant contexts from the memory. Considering the sparsity of attention score matrices, processing each token typically requires only a small portion of its contexts (Zhang et al., 2023b), and the remaining irrelevant contexts act as noise, leading to attention distraction issues (Tworkowski et al., 2023). We thus construct an external memory containing distant context information. Only relevant information within the memory is selected for each computation step, and other irrelevant noises are ignored. Owing to this, LLMs can understand whole long sequences using a finite-size window and avoid noisy contexts.
The vast amount of noisy context tokens in long sequences poses significant challenges to effective and efficient memory lookup. To address these challenges, we design a block-level context memory mechanism. Specifically, InfLLM organizes past key-value vectors into blocks, each containing continuous token sequence. Within each block, the semantically most significant tokens that receive the highest attention scores are selected as the unit representation for subsequent relevance computation in memory lookup. This design offers two primary benefits: (1) Effective Lookup: The coherent semantics of each block can more effectively fulfill the requirements for relevant information retrieval compared to single tokens. The selection of unit representations minimizes the interference of unimportant tokens in relevance computation, enhancing the overall hit rate of memory lookup. (2) Efficient Lookup: The block-level memory unit eliminates the need for per-token relevance computation, significantly reducing computational costs. Moreover, block-level units ensure contiguous memory access, thus minimizing memory loading costs and enhancing computational efficiency. Furthermore, considering the infrequent usage of most units, InfLLM offloads all units on CPU memory and dynamically retains the frequently used units on GPU memory, significantly reducing GPU memory usage. Notably, the block-level memory mechanism in InfLLMM does not involve any additional training, and can be directly applied to any LLMs.
To evaluate the effectiveness of InfLLM, we employ Mistral-7B-inst-v00 (Jiang et al., 2023) and -lama-3-8B-Instruet (Meta, 2024) as base models, which are pre-trained on the sequences containing no more than 32K and 8K tokens. We use two widely-used benchmarks, x--Bench (Zhang et al., 2023a) and Longbench (Bai et al., 2023), for evaluation. Especially, the average sequence length in oo-Bench exceeds 100K tokens, which is challenging for most existing LLMs. Compared to typical methods that continually train LLMs on longer sequences, the experimental results demonstrate that InfLLM enables the LLMs pre-trained on the sequences containing few thousand tokens to achieve comparable performance without any additional training. Moreover, we examine InfLLM on the sequences containing 1, 024K tokens, and InfLLM can still effectively capture long-distance dependencies, demonstrating the potential of InfLLM in scenarios involving long streaming inputs.
2 Related Work
Enabling LLMs to process long sequences has been extensively studied (Dong et al., 2023; Tay et al., 2023; Huang et al., 2023) and can generally be categorized into two main approaches: context length extrapolation and efficient context computation. The former aims to enable LLMs trained short sequences to process much longer sequences. The latter focuses on enhancing the computational efficiency of attention layers, allowing efficient pre-training LLMs from scratch to process longer sequences. Although the focus of this paper is context length extrapolation, we also detailedly introduce efficient context computation. We also present the relevant works for memory- based models.
Context Length Extrapolation. Due to the high computational and memory requirements, the training of L.LMs is often restricted to short sequences. Directly applying L.MMs to long sequences will suffer from out-of-domain and distraction challenges caused by lengthy and noisy inputs (Han

---

et al., 2023; Tworkowski et al., 2023). Consequently, context length extrapolation has garnered attention as method improve the sequence length for LLMs without incurring additional training. The earliest approaches involve designing new relative positional encoding mechanisms during pre-training (Press et al., 2022; Sun et al., 2023). Subsequent studies mainly focus on the widely-used rotary position embedding (RoPE) (Su et al., 2021), and propose to achieve length extrapolation by downscaling or reusing the original position indices (Chen et al., 2023b; Peng et al., 2023; Chen et al., 2023a; Jin et al., 2024; An et al., 2024). These works can alleviate the out-of-domain issue from the unseen length, but can not alleviate the distraction challenge of noisy contexts. To address this, Xiao et al. (2023) and Han et al. (2023) employ the sliding window attention mechanism and directly discard all distant contexts to streamingly read extremely long sequences. However, as these models overlook information from distant tokens, they not capture the oong-iistance dependencies for long-text understanding. In this paper, InfLLLM utilizes the sliding window attention mechanism, and additionally constructs an efficient context memory provide LLMs with relevant context information, enabling LLMs to effectively read and understand extremely long sequences.
Efficient Context Computation. The quadratic computational complexity of the attention layers is capabilities of LLMs. Thus, numerous scholars have endeavored to design efficient attention mechanisms, including the utilization of sparse attention (Zaheer et al., 2020; Beltagy et al., 2020; Child et al., 2019; Ainslic et al., 2020; Zhao et al., 2019), approximating attention computations using kernel functions (Kitaev al., 2020; Wang et al., 2020; Katharopoulos et al.. 2020), and replacing the attention layer with linear- complexxy state-space models (Gu et al., 2022; Gu & Dao, 2023). These approaches necessitate modification in the model architecture, requiring retraining the models. Simultaneously. many researchers enhance the inference efficiency by evicting useless key-value vectors reduce computation (Zhang et al., 2023b; Li et al., 2024; Ge et al., 2023). These methods can not extrapolate the context window of LLMs without further training due to out-of-domain issues caused by unseen positions. Recently, some researchers begin to explore the intrinsic sparse attention patterns of long-context LLMs and discard the redundant attention computation for acceleration (Jiang et al., 2024).
a primary factor limiting the lengthy
Memory-based Models. Memory networks have been studied for decades, which are proven effective in providing models with additional knowledge and information storage capabilities (Graves et al., 2014; Weston et al., 2015; Sukhbaatar et al., 2015; Miller et al., 2016). With the success of pre- trained models, memory layers have also been gradually applied in the training processes of recurrent transformer layers, enabling models to process long sequences recursively (Dai et al., 2019; Rae et al., 2020; Khandelwal et al., 2020; Wu et al., 2022; Bertsch et al., 2023; Munkhdalai et al., 2024). These works split sequences into segments, encoding each segment individually, and use memory to store context information from preceding segments. While these approaches are similar in concept to InfLLM, they involve modifications to the model architecture and requires further training the whole model. Besides, most existing memory- based methods focus on token-level memory units (Wu et al., 2022; Bertsch et al., 2023), which require lot of time to build retrieval indexes for large-scale tokens in each input long sequence. Some methods also adopt block-level memory (Mohtashami & Jaggi, 2023; Tworkowski et al., 2023), these methods highlight the process of training effective block representations with long sequence data. In contrast, we aim to explore the inherent characteristies of LLMss and propose training-free memory module for long-text understanding.
3 Methodology
As shown in Figure 1, InfLLM builds a training-free context memory to efficiently provide highly- relevant contexts for each token, endowing the sliding window attention mechanism with the ability to capture long-distance dependencies.
3.1 Overall Framework
The main restrictions for improving the length generalizability of LLMs come from the out-of -omaan and distraction issues caused by the lengthy and noisy contexts. To address these, following previous works (Xiao et al., 2023; Han et al., 2023), we adopt the sliding window attention mechanism, which only considers local tokens for each step. Additionally, we construct an extra context memory module to provide relevant context information to capture long-distance dependencies.
3

---

Memory Units
Representative
Tokens
Q Lookup
LLM
Context
Memory
Current Key-Value Cache
Inputs
Past Key-Value
Initial Tokens
Evicted Tokens
Local Takens
Current Tokens
Figure The illustration of InfLLM. Here, the current tokens refer to tokens that need be encoded in the current computation step. The past key-value vectors can be divided into the initial tokens, evicted tokens, and local tokens, arranged the furthest to the nearest relative to the current tokens. For each computation step, the context window consists of the initial tokens, relevant memory units, and local tokens.
Specifically, we denote the long input sequence as 8 (41)=11 Due to the limited GPU memory, instead of encoding the whole at once, we encode the input sequence s chunk-by-chunk and generate the output token-by-token. For each computation step, the inputs consist of past key-value vectors {ti+tp ) For encoding steps, (x equals the chunk
P = [kk, v and current tokens X
size, and for decoding steps, 1x equals one.
According the distances from current tokens, we can divide P into three groups: initial tokens, I= P [1:ty]- evicted tokens, E P(tr+1pp=11))- and local tokens, L = P [Ip-Iz+1dp)) arranged from the furthest the nearest relative the current tokens. Here, 1p, 4,, It. refer the length of past key-value vectors, initial tokens, and the local window size. All evicted tokens, E, are stored in the context memory, consisting of multiple memory units. For each step, InfLLM concatenates the initial tokens, relevant memories units from context memory, and local tokens to form the current key-value cache, C Concat(I, f(X, E), _). f(() refers the lookup operation of context memory. The attention output is calculated as:
Attn QQX, Concat(Ca, KX), Concat(C,, VX)].
Here, Q, K, and V are parameters in attention layers, Ca and C,, refer to the key and value vectors in C. If f()) always returns empty sets, InfLLM is degenerated into LM-Infinite (Han et al., 2023) and Streaming-LLM (Xiao al., 2023), which directly discards distant contexts.
3.2 Context Memory
Previous findings indicate that the attention score matrices of LLMs are sparse, and we can generate the same outputs with only small portion of key-value vectors preserved (Zhang et al., 2023b). Inspired by this, we design a context memory to efficiently look up relevant contexts from large-scale evicted tokens and ignore irrelevant ones to computational costs. The most intuitive way is to construct memory consisting of token-level memory units for every past key-value vectors, and every attention head separately, which would result in massive memory units, unacceptable computation, and non-contiguous memory access costs. Thus, considering the local semantic coherence of long sequences, we split the past key-value vectors into blocks, each serving as a memory unit, and conduct memory lookup at the block level to reduce the costs while preserving the performance.
In this subsection, we will introduce the details of the block-level memory units. Then we present the method to assign positional embeddings for selected relevant memory units and cache management for the context memory.
Block-Level Memory Units. Block-level memory units can save computation costs compared to token-level ones. It also poses new challenges for unit representations, which are supposed to contain the semantics of the entire unit for effective relevance score computation and memory-efficient for context length scalability. Traditional methods usually involve training additional encoder

---

to project a given unit into low-dimension vector. Inspired by the token redundancy in hidden states (Goyal al., 2020; Dai et al., 2020), we select several representative tokens from the entail blocks as the unit representation. For the m-th token, we define the representative score as:
1 L
I'
Am+j
km
where qm-j is the query vector for (m + j)-th token and km is the key vector m-th token. Intuitively, I'm represents the significance of the m-th token its corresponding local window, indicating the extent of its influence on other tokens within the local window. The computation of representative scores requires no additional parameters.
Formally, given the evicted tokens, E, we split it into several memory units, each containing l. tokens. For each unit, the rı tokens with the highest representative scores are selected as representative tokens. Generally, rh small positive integer. Let us denote memory unit as B [(k]. vj )) and the representative tokens of this unit as R(B) = {kk, vo, 3
For the memory lookup phrase, only km units with the highest relevance scores are loaded for the current attention computation. We calculate the relevance score between B and current tokens X as:
2x
Fk
sim X, B)
qitt
k
Notably, the representative tokens selection is training free method to obtain the unit representations. Here, we can also train an additional encoder to generate more expressive unit representations, which we leave for future work.
Positional Encoding. Existing LLM training usually employs a finite number of positional encodings, which encounter out-of-domain distribution challenges when directly applied to longer sequence processing (Han et al., 2023). Besides, InfLLM, the current key-value cache composed some discontinuous text blocks, and directly assigning continuous positional encodings them would also lead to mismatch issues and confuse the model. Therefore, inspired by previous works (Raffel et al., 2020; Su, 2023), assign all tokens beyond the local window size with the positional encodings. Specifically, the distance between tokens in context memory units and current tokens is set as lz.
Cache Management. To enable LLMs to process extremely long sequence streams while capturing the semantic relevance contained in the long contexts, we need to retain all memory units and look up them at each computation step. Considering the infrequent usage of most units, we employ an offloading mechanism, storing most memory units in CPU memory and only preserving the representative tokens and memory units needed current steps in GPU memory. Additionally, given the semantic coherence of long sequences, where adjacent tokens often require similar memory units, we allocate eache space in GPU memory, managed using a least recently used strategy. This approach allows for efficient encoding of extremely long sequences using limited GPU memory. From the observation, our offloading mechanism enables InfLLM to process sequences consisting of 100K tokens with only 26G VRAM. Besides, the miss rate of our GPU cache is quite low, which means the offloading mechanism does not introduce significant time overhead in memory loading while saving GPU memory usage. The details can be found in the Appendix.
Furthermore, for extremely long sequences, the representative tokens of each unit can also be offloaded to the CPU memory, constructing efficient k-nearest-neighhor index, and thereby further reducing computational complexity.
Experiments
4.1 Settings
Datasets. We adopt representative tasks in a widely-used long document benchmark, co- Bench (Zhang et al., 2023a) for evaluation. We adopt the English datasets for evaluation as the base models are mainly pre-trained on English corpus. The datasets in Bo-Bench cover diverse tasks
5

---

Table 1: The results of InfLLM and baseline models co-Bench. The 95% quantile for text lengths in co-Bench is 214K. The context window size for sliding window models refers to the local window size, and for InfLLM refers to "local window size + selected memory size".
Window
Streaming
R.PK
R.Num R.KV
Choice
QA
Sum
Math.F
Avg.
Mistral-based Models (7B)
Mistral
32K
28.8
28.8
14.8
44.5
12.9
25.9
20.6
25.2
NTK
128K
100.0
86.8
19.2
40.2
16.9
20.3
26.9
44.3
SelfExtend
128K
100.0
100.0
15.6
42.8
17.3
18.8
19.1
44.8
Infinite
32K
28.8
0.4
42.8
11.4
16.3
21.6
Streaming
32K
28.8
28.5
0.2
42.4
11.5
22.1
16.9
21.5
H2O
32K
8.6
4.8
2.6
48.0
15.6
24.4
26.9
18.7
InfLLM
16K
100.0
96.1
96.8
43.7
15.7
25.8
25.7
57.7
Llama- based Models (8B)
Llama-3
8K
8.5
7.8
6.2
44.1
15.5
24.7
21.7
18.4
NTK
128K
0.0
0.0
0.0
0.0
0.4
6.4
2.6
1.3
SelfExtend
128K
100.0
100.0
0.2
19.7
8.6
14.7
22.6
38.0
Infinite
8K
6.8
7.6
0.2
41.5
14.6
20.8
20.6
16.0
Streaming
8.3
0,4
40.6
14.3
20.4
21.4
16.3
H2O
8K
2.5
2.4
0.0
0.0
0.7
2.8
6.0
2.1
InfLLM
8K
100.0
99.0
5.0
43.7
19.5
24.3
23.7
45.0
including question answering, summarization, context retrieval, and mathematic computing. The average length for co-Bench 145.1K. The 95% quantile for sequence lengths is 214K, which is far beyond the maximum length of the base models. Detailed statistics and task descriptions these datasets are listed in the Appendix. Besides, we also conduct evaluation LongBench (Bai et al., 2023). The results for LongBench can be found in the Appendix.
Baseline Models. To verify the effectiveness of our proposed method, we compare InfLLM with the following competitive baseline models: (1) Original models: we present the performance of the original LLMs without context length extrapolation. (2) Position downscaling and resuing: NTK- aware scaled RoPE (NTK) (LocaILLaMA, 2023) designs a nonlinear interpolation method, which basically changes the rotation base of RoPE. SelfExtend reuse the position ids across neighboring tokens, which makes the extended relative positions in the scope of the training context window. (3) Sliding window: these methods apply the sliding window mechanism to discard distant contexts, including Infinitf (Infinite) (Han et al., 2023) and Streaming (Stream) (Xiao et al., 2023). Therefore, for each attention computation step, the input length does not exceed the context window. (5) Key-value eviction: KV eviction methods aim to discard useless key-value vectors during long sequence processing and thus usually used reduce the computation complexity. We present the results of a widely-used key-value eviction method, H2O (Zhang et al., 2023b). The key-value eviction method cannot generalize to longer sequences due the unseen position embeddings and expected to achieve unsatisfactory performance.
Here, InfLLM and the models with the sliding window mechanism be used process extremely long streaming inputs. For NTK and SelfExtend, we extend the context window to 128K, which enables LLMs to process most instances in oc-Bench.
4.2 Implementation Details
In this paper, aim to enable LLMs trained with limited sequence length to read and understand extremely long sequences without further training. We adopt Mistral-7B-Insttuee I-sttrruutt--0.2 (Jiang et al., 2023) and Saamma---SB-Insttie 7B-Instructt-0.2 and Llama-3-88 3------------uuu-
(Meta, 2024) as base models. The maximum length of Mistral-
32K and 8K, respectively.
For our model, we set the encoding chunk size as 512, and the memory unit size for past key-value vectors, 4ns, as 128. The number of representative tokens, Tk. set as For both Mistral-based and Llama-3-based InfL.LM, we set the local window size as 4K. For Mistral-based InfL.LM, we load 96 relevant memory units for each step, and for Llama-3-based InfL.LM, we load 32 relevant memory units. The number of initial tokens is set 128 for LM-Infinite, StreamingLLM, and InfLLM to

---

1048k Table 2: The comparison between InfLLM and models with continual pre-training, Llama-3-8B Prrrucc-rradient-e (Llama-1M) 1M). InfLLM can achieve comparable performance with Llama- with less computation consumption and memory usage.
Train-Free
R.PK
R.Num
R.K.
Choice
QA
Sum
Math.F
VRAM
Time
Llama-IM IM InfLLM
X 100.0
99.8
23.2
51,5
13.6
18.5
18.3
76.6G
40.4s
/
100.0
99.0
5.0
43.7
19.5
24.3
23.7
26.3G
26.7s
x 100.0
100.0
55.8
39.3
20.3
17.1
31.4
26.3G
26.7s
cover the system prompts and task descriptions. We adopt FlashAttention (Dao, 2023) accelerate experiments for all baseline models. Please refer to the Appendix for more details.
4.3 Main Results
The results for Mistral-based models and Llama-3-based models are reported in Table From the results, we can observe that: (1) Compared to models with the sliding window mechanism, which can also read extremely long sequences, our method demonstrates a significant performance improvement. This indicates that the context memory in InfLLM accurately supplement LLMs with relevant contextual information, enabling efficient and effective understanding and reasoning on long sequences. (2) The position downscaling and resuing methods, NTK and SelfExtend, tend to compromise model performance while extending the sequence length to 128K. That is because these models cannot address the distraction issue caused by noisy contexts. In contrast, can consistently enhance performance for extremely long sequences. We successfully generalize Llama-3 from 8K length to more than 16 times its length, achieving commendable performance on the co- no-Bench. (3) The position downscaling and resuing methods can increase the maximum sequence length of LLLSs but also raise the computational and memory costs, limiting these methods' application. In contrast, Inf utilizes block-level memory and offloading mechanism, enabling efficient processing of long sequences within limited resources.
model
4.4 Comparing Models with Continual Training
In this paper, focus expanding the context window of LLMs without additional training. In this section, we compare InfLLM with models that undergo continual training on long sequences in terms of both performance and efficiency. Specifically, we select Llama-3-8B-Instee (Llama- IM)', which have been further fine-tuned on long-text data and chat datasets, extending its context window 1048K. Besides, we also employ InfLLM on the Llama- 1M, where we set the local window as 4K and selected memory size as 4K. We present the results on cc-Bench, the GPU memory usage, and time consumption in Table 2. From the results, we can observe that: (1) Compared to models that have undergone continual training on long sequences, InfL.M can achieve comparable or even superior results without any additional training. This suggests that LLMSs inherently possess the capabiliiy to identify key information in long sequences and to understand and reason effectively. Notably, Llama-IM requires 512 GPUs for continual training, which is unaffordable for many researchers. In contrast, InfLLM does not require any training, which indicates the practicability of InfLLM. (2) In terms of efficiency, InfLLM achieves 34% decrease in time consumption while using only 34% of the GPU memory compared to the full-attention models. Moreover, at longer sequence lengths of 256K tokens, the full-attention baseline fails due to out- out-of-ment of-momorroo errors, while InfLLM can efficiently process sequences up to 1024K tokens on single GPU. (3) InfLLM can also be directly combined with the model with continual training and achieve comparable or even superior results with only 8K context window. It indicates that Infl.LM can also serve as an efficient way to improve the inference speed.
4.5 Comparing to Retrieval-Angmented Generation
InfLLM leverages the intrinsic capacity of LLMs construct context memory for gathering token-relevant information, a concept similar to retrieval augmented generation (RAG) (Lewis et al., 2020; Nakano al., 2021). However, compared to using RAG, where historical contexts are treated
https://ttggiigfiee..eograeaeiii ---taaam---------stttt-t-----e- EnstructGrraiiettio 1048k
7

---

26-
Rettieee KV Performance (9r)
24-
Retriee KV Performance (7))
24-
12-
Retrieve KV Performanse
24
Performance (%)
12-
Performance
Performance
Math.F
18-
QA
16
16-
3
20
46
28
56 100
130
300 Memory Unit Size
300
159
250
# Representative Tokens
Selected Units
(a)
(b)
(c)
Figure 2: Extra studies about InfLLLM. Here, (a), (b), and (c) investigate the impact of the context memory under different numbers of representative tokens, different numbers of selected units, and memory unit sizes, respectively.
as a searchable database for long-sequent understanding (Xu et al., 2023), InfL.LM has several advantages: (I) Training-Free: RAG requires additional retrieval data to train retrieval model, whereas InfLLM is training-fived frree and applicable to any LLMs. Besides, RAG also necessitates fine- tuning LLMs to adapt to the inputs augmented by the retrieved knowledge. (2) Broader Applicability: RAG models are usually limited by the performance of their retrieval components. Besides, existing retrieval models will suffer from of-dissribuutinn issues, struggling perform well on tasks outside their training distribution (Lin et al., 2023; Muennighoff et al., 2023). This limitation adversely affects the overall performance of the RAG system. contrast, InfLLM has no specific requirements for tasks and can be feasibly used for long sequences.
To verify the generalization capabilities of InfLLM, conduct experiments to comparing RAG and InfLLM on three context retrieval tasks. We utilize ES-mistral-7B-innttue (Wang et al., 2024b) as the retrieval model. The results are shown in Ta- ble Our findings demonstrate that even without additional data or training, InfLLM can consistently outperform RAG models, underscoring its superior generalization capabilities. The dependency on an external retrieval model makes RAG less flexible handling diverse tasks.
Table The comparison between InfLLM and RAG.
Task
R.PK
R.Num
R.KV
RAG-E5 InfL.LM
89.2
65.4
13.2
100.0
96.1
96.8
4.6
The Impact of Memory Settings
InfLLM relics on the context memory to look up relevant information. We further explore the impact of core components in the context memory, specifically the representative tokens and memory units. The results are shown Figure 2.
Different Number of Representative Tokens. InfLLM splits key-value vectors into memory units and selects several representative tokens from the unit to serve the unit representations. Consequently, the ability of these representative tokens semantically represent the entire unit directly impacts the model's performance. We conduct experiments with the number of representative tokens (1, 2, 4, 8). The results are shown in Figure 2a. It is observed that as the number of representative tokens increases, there is a trend of improvement in the model performance, which indicates that more representative tokens tend better represent the semantic content of the memory units. However, is noted that when the number of representative tokens reaches 8, there is a slight performance decrease. This decline can be attributed to the inclusion of semantically irrelevant tokens as unit representations. More efficient and powerful unit representations will further enhance model performance for future work.
Different Number Selected Units. The selected units are utilized to provide relevant context to LLMs. We conduct experiments with the number of units set {2, 4,,4,8, 8, 16,32, 64, 96, 128). From Figire 2b, we can observe that as the number of selected units increases from to 32, the model performance significantly improves, which is attributed to that more units imply greater recall rate

---

of relevant content. Larger unit quantity also leads to an increase in the required memory scheduling time and the computational time for attention. Therefore. further enhancing lookup accuracy remains a crucial direction for improving the efficiency of InfLLMM.
Different Memory Unit Size. Each memory unit is supposed be coherent semantic unit. Exces- sively large unit sizes can hinder precise lookup, while a small size will increase the computational overhead of memory lookup. We evaluate InfLLM with the unit size as {32, 64, 128, 256} and keep the total context length as 12K. The results are shown in Figure 2c. It can be observed that the optimal unit size varies for different tasks due the varying characteristics of input sequences. For example, in Retrieve.K KV, a key-value pair constitutes a semantic unit, while in Math.Find a single number represents a semantic unit. Employing heuristic rules to segment context can easily lead to suboptimal performance. Therefore, exploring how to dynamically segment context is an important direction for future research.
4.7 Ablation Study
To further verify the effectiveness of dynamic memory lookup and unit representations, we conduct ablation studies in this section. The results are shown in Table 4.
Context Memory Lookup. InfLLM adopts dynamic context memory lookup for both input encoding and output decoding steps for comprehensive long-text understanding. We present the results of InfLLM with only lookup in output decoding (Decoding-Onig and without any memory lookup (w/o Lookup). It can be observed that significant decline in model performance is associated with a reduction in the number of memory lookup iterations. This indicates that distant contextual information is crucial for both phases. The model requires the integration of long-
Table 4: The results for ablation study.
Task
R.KV
Math.F
QA
InfLLM
96.8
25.7
15.7
Decoding-One
85.2
26.3
12.0
w/o Lookup
0.4
16.3
11.4
Mcan Repr
84.6
25.1
14.9
the punginput encoding and auswer-generaticn distance context to generate a coherent context memory for input understanding. LLM is supposed to collect useful information from massive past context information to generate the correct answers.
Unit Representation. We design a block-level memory for efficient context information lookup. We select several representative tokens as the unit representations for relevance computation. We present the results of InfLLM with another training-free ffree representation method (Mean Repr), which computes the representation by averaging the key vectors in a memory unit. From the results, we can observe that InfLLM with average representations can also present competitive performance. It indicates that the original attention vectors in LLMs are effective for relevance score computation, and exploring more efficient unit representations is an important future direction.
4.8 Scaling to 1,024K Context
To assess the effectiveness of InfLLM on extremely long sequences, this subsection, we scale the se- quence length to 1024K to evaluate the capacity of InfLLM capture contextual relevance in long se- quences. Specifically, we adopt the Retrieve.Passke task in oc-Bench for evaluation. This task prompts LLMs to find 5-digit sequence among lengthy and noisy contexts, which requires LLMs to lo- cate relevant information among long sequences ef- fectively. We automatically generate inputs with {32, 64, 128, 256, 512, 768, 1024] thousand tokens and for each length, we generate 50 instances for evaluation. We adopt Mistral as the base model.
Detrieve.PassKer
100
Accuracy (%)
LM-Infinite
Ours
200
400
600
800 1000
Length (K)
Figure 3: The results on sequences with dif- ferent lengths.
The results are shown in Figure 3. From the results, we can observe that InfI.LM can accurately locate the key information from length noises and achieve 100% accuracy even when the context length scales to 1024 thousand tokens. However, LM-Infinite only attend to the tokens within the local window,

---

which leads to rapid decline in its performance as the sequence length increases. It proves that InfLLM can accurately capture the long-distance dependencies for effective long-sequence reasoning.
5 Conclusion
In this paper, we propose training-free method to improve the length generalizability of LLMs. Based on the sliding window attention mechanism, we construct an additional context memory module, which can help LLMs select relevant information from massive contexts to capture long- distance dependencies. The experiments on two widely-used long-text benchmarks show that InfL.LM can effectively improve the ability of L.Mss, which are trained on sequences with few thousand tokens, to process extremely long sequences. In the future, we will explore efficient training of the context memory module to further enhance the model performance. Besides, combining the key-value cache compression methods with InfLLM can further reduce the computational and memory costs. We hope InfLLM can boost the development of streaming applications of LLMs.
Acknowledgement
This work supported by the National Key R&D Program China (N..2022D011050)) Quan Cheng Laboratory (Grant No. QCLZD202301) and Institute Guo Qiang at Tsinghua University. Pengle Zhang is supported by Tsinghua University Initiative Scientific Research Program (Student Academic Research Advancement Program).
References
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. ETC: encoding long and structured inputs in transformers. In Proceedings of EMNLP, pp. 268-284, 2020.
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free free long-context scaling of large language models. CoRR, abs/2402.17463, 2024. doi: 00.48550/ARXI..440.. 17463.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang. Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308. 14508, 2023.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150. 2020.
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. Unlimiformer: Long-range transformers with unlimited length input. CoRR, abs/2305.0162 2023.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.077 002288. 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of NeurIPS, 2020.
10

---

Guanzheng Chen, Xin Li, Zaiqiao Meng. Shangsong Liang, and Lidong Bing. CLEX: continuous length extrapolation for large language models. CoRR, abs/2310. 2023a. doi: 10.48550/ ARXIV.2310.16550
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306. 15595, 2023b.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality, March 2023.
Rewon Child, Scott Gray, Alec Radford, and Hya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/ 190//990..1000 10509, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of ACL, pp. 2978-2988. Association for Computational Linguistics, 2019.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Hunnel-transformer Filtering out sequential redundancy for efficient language processing. In Proceedings of NewrIPS, 2020.
Tri Dao. Flashattention-2: 2: Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691, 2023.
Yiran Ding, Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang. Jiahang Xu, Fan Yang. and Mao Yang. Longrope: Extending LLM context window beyond 2 million tokens. CoRR, abs/2402. 13753, 2024. doi: 10448500AAXX1.2200.11555
Zican Dong, Tianyi Tang, Junyi Li, and Wayne Xin Zhao. survey on long text modeling with transformers. CoRR, abs/2302. 14502, 2023.
Danny Driess. Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Proceedings of ICML, volume 202 of Proceedings of Machine Learning Research, pp. 8469-8488. PMLR, 2023.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for lmss. CoRR, 10.023 2023. doi: 00.45550/ARXXV2231000822
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating BERT inference via progressive word- vector climination. In Proceedings of ICML, volume 119, pp. 3690-3699, 2020.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CORR, abs/2312.00 has/2312.00752. 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Proceedings of ICLR. OpenReview.net, 2022.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. CoRR, abs/2308. 16137, 2023.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlic Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song. Jie Tang. Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. Pre-trained models: Past, present and future. Al Open, 2:225-250, 2021.
Yunpeng Huang, Jingwei Xu, Zixu Jiang, Junyu Lai, Zenan Li, Yuan Yao, Taolue Chen, Lijuan Yang, Zhou Xin, and Xiaoxing Ma. Advancing transformer architecture long-context large language models: A comprehensive survey. CoRR, 0tt//211..1255.1 2023.
11

---

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford. Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR,
2023.
Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, al. Minference: Accelerating pre-filling for long-context llms via dynamic sparse attention. In Workshop on Efficient Systems for Foundation Models @@@ ICML,2024, 2024.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. LLM maybe longlm: Self-extend LLM context window without tuning. CoRR, abs/2401.0133 001325, 2024.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of ICML, volume 119, pp. 5156-5165. PMLR, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proceedings of ICLR. OpenRe- view.net, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proceedings of ICLR. OpenReview.net, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of SOSP, pp. 61 1-66. ACM, 2023. doi: 10.1 145/36660000.6 3613165.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmennte generation for knowledge-intensive NLP tasks. In Proceedings of NeurlPS, 2020.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma. and Hao Zhang. How long can open-source Ilms truly promise context length?, June 2023.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXiv preprini arXiv:2404. 2ete:2204.44466 14469, 2024.
Jacky Liang Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. In Proceedings of ICRA, pp. 9493-9500. IRRE, 2023.
Sheng-Chied Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen- tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards gen- eralizable dense retrieval. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of ACL: EMNLP, pp. 6385-6400. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. 1...65551220....HINGEEEEEEEE
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. CoRR, ab</2307.03172. 2023.
LocalLLaMA. Ntk-aware scaled rope allows Iaama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. 2023.
Meta. Introducing meta laaaa 3: The most capable openly available 1lm to date. 2024.
12

---

Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hosseni Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. Proceedings of EMNLP. pp. 1400-1409. The Association for Computational Linguistics, 2016.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB: massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.). Proceedings of EACL, pp. 2006- 2029. Association for Computational Linguistics, 2023. doi: 10.
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. CoRR, abs/2404.07143, 2024. doi: 10.48550/
AR471.2404.07143.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question -answerng with human feedback. CoRR, abs/2112.099332, 2021.
OpenAL. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of NeurlPS, 2022.
Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of UIST, pp. 2:1-2:22. ACM, 2023.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context window extension of large language models. CoRR, abs/2309.00071, 2023.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of ICLR. OpenReview.net, 2022.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307. 007924, 2023.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloc Hillier, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling. In Proceedings of ICLR. OpenReview.nee, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21: 40:1-140:67 2020.
Jianlin Su. Rectified rotary position embeddings, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proceedings of NewrlPS, pp. 1440-2448, 2015.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatal transformer. In Proceedings of ACL, pp. 14590- 14604. Association for Computational Linguistics, 2023. doi: 10.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: survey, ACM Comput. Surv. 15:1:1099, 2023,
13

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. Focused transformer: Contrastive training for context scaling. CoRR, abs/2307.031700 2023.
Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey large language model based autonomous agents. Frontiers Comput. Sci., 18(6): 18635,, 2024a. doi:
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. CoRR, fass/2001.00360 2024b. doi: 10.48550/
ARX/V.2401.0035. .00368.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of ICLR, 2015.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins. and Christian Szegedy. Memorizing trans- formers. In Proceedings of ICLR. OpenReview.net, 2022.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. CoRR, abs/2309. 17453, 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. CoRR, abs/2309.16039. 2023.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. CoRR, abs/2310.03005 2023.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Proceedings of NeurlPS, 2020.
Xinrong Zhang. Yingfa Chen, Shengding Hu, Qihao Wu, Junhao Chen, Zihang Xu, Zhenning Dai. Xu Han, Shuo Wang, Zhiyuan Liu, and Maosong Sun. Infinitehench: 128k long-context benchmark for language models, 2023a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. Hoo: Heavy- hitter oracle for efficient generative inference of large language models. CoRR, abbs/2306.14048, 2023b.
Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. Explicit sparse transformer: Concentrated attention through explicit selection. CORR, abs/1912.11637 2019.
Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bing Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding. CoRR, abs/23312.17044 2023.
14

---

Broader Impact
This paper presents work whose goal is to advance the field of long sequence processing for large language models. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Limitations
In this paper, we propose InfLLM, a method for extending the context window of LLMs without additional training. We verify the effectiveness of our model using widely-used long-text evaluation benchmark oc-Bench. However, our method still has the following limitations: (1) We store large amount of past key-value (KV) cache in the CPU memory, which increases CPU memory usage. In the future, we can reduce CPU memory requirements by integrating techniques like KV cache quantization. (2) While InfLLM reduces the computational overhead for processing long texts in LLMs, there still room for speed-up. In the future, we can further enhance the inference speed of InfLLM by integrating it with inference frameworks like llama.cpp' and vllm (Kwon et al., 2023).
A
Cache Management Strategy
Due to the massive amount of memory units for extremely long sequences, we adopt an offloading mechanism to save GPU memory costs. Considering the infrequent usage of memory units, we offload most memory units to CPU memory and only preserve the frequently used memory units and current needed memory units in the GPU mem- ory. To this end, maintain cache in GPU memory effectively utilize GPU memory and reduce the commu- nication between CPU and GPU. The size for our GPU cache is fixed, and therefore we design a least recently used (LRU) strategy for cache management. In this sec- tion, we will introduce the management strategy in detail.
Missing Rato
LRU
5%
FIPO
4%
3%
0.0
0.
0.4
0.5
0.8
1.0
Soore Decay
Figure 4: Missing rates of different cache management strategies.
Loading Memory Units For each computation step, we
first compute the relevance scores for each memory unit to determine which units should be used. Then, for each needed memory unit, first search it in cache. If there is no hit, then we proceed with the transfer from CPU memory to GPU memory.
Offlodding Memory Units After the attention computation, we need to offload redundant memory units to keep the GPU cache fixed. To this end, we apply an LRU strategy. Specifically, for each memory unit loaded into our GPU cache, we assign frequency score 8 for it, which will be used to determine whether this unit should maintained the GPU cache or offloaded to CPU memory to save GPU memory costs. The frequency scores are updated after the attention computation. Specifically, we update the score as follows:
8b = -b.bd. d+ attention aitention_score(qjgr)) score( k;);
where L, represents the number current tokens involved in this lookup, attention attention_score(q. k) denotes the attention score between Q with respect to k (ranging from 0 to 1) obtained after performing the attention computation. d is hyper-parameter representing the decay coefficient, used to incorporate the influence of previous lookups. After each attention computation, we sort all the memory units in our GPU cache according to their frequency scores 8t, and offload the units with the lowest scores back to the CPU memory.
To verify the effectiveness of our cache management strategy, we evaluate the cache missing rate of different cache management strategies sample of data from the GovReport dataset. Specifically, we compare our LRU strategy with (1) Random: randomly selecting units from the GPU cache to
15

---

Table 5: The results of InfLLM and baseline models LongBench. The 95% quantile for text lengths LongBench is 31K. The context window size for sliding window models refers to the local window size, and for InfLLM refers to "local window size selected memory size".
Window
NQA
Qasper
MFQA
HQA
2WikiMQA
Musique
Mistral-based Models (7B)
Mistral
32K
22.06
29.16
47.65
37.53
21.96
19.03
Infinite
6K
18.44
30.02
39,05
32.02
22.27
15.81
Streaming
6K
17.92
30.05
39.09
32.18
21.83
14.71
InfT.LM
6K
22.12
29.33
47.42
36.56
22.31
17.68
InfL.LM
12K
23.03
29.52
47.62
39.53
23.61
18.92
Hama---based Models (8B)
Llama-3
8K
19.85
42.36
41.03
47.38
39.20
22.96
Infinite
8K
19.39
42.80
40.44
43.77
37.89
18.33
Streaming
8K
20.05
42.46
39.54
43.69
37.89
19.68
InfLLM
8K
22.64
43.70
49.03
49.04
35.61
26.06
Window
GovReport
QMSum
MultiNews
TREC
TQA
SAMSum
Mistral-based Models (7B)
Mistral
32K
31.12
23.87
26.62
71.00
85.97
42.29
Infinite
6K
29.74
21.92
26.65
70.00
85.22
41.60
Streaming
6K
29.83
21.94
26.64
70.00
85.57
41.31
InfLLMM
6K
31.03
23.49
26.70
69.00
86.67
42.52
InfLLM
31.37
23.77
26.66
71.00
87.34
41,80
Llama-3-based Models (8B)
Llama-3
8K
29.94
21.45
27.51
74.00
90.50
42.30
Infinite
8K
29.25
21,41
27.62
74.00
90.08
41.72
Strcaming
8K
29.17
21.33
27.56
73.50
90.08
41.55
InfLLLM
8K
30.76
22.70
27.57
73.50
90.91
42.43
Window
PsgCount
PsgRetrieval
LCC
RepoBench-P
Avg.
Mistral-bases Models (7B)
Mistral
32K
3.95
86.94
57.42
54.14
43.78
Infinite
6K
2.08
42.80
57.12
53.43
39.07
Streaming
2.50
42.17
55.38
$1.46
38.67
InfLLMM
6K
2.87
64.00
56.67
52.97
41.90
InfLLM
12K
3.01
87.42
56.69
52.09
44.02
Llama-3 Babsed based Models (8B)
Llama-3
SK
8.50
62.50
60.83
49.14
44.73
Infinite
8K
4.50
55.000
60.12
48.62
43.03
Streaming
8K
5.00
49.00
60.35
48.95
42.99
InfT.I.M
8K
7.17
84.00
59.88
46.48
46.95
offload. (2) First-infirst-out (FIFO): offload the unit that is first loaded in the GPU cache. The results are illustrated in Figure 4. It is observable that the LRU strategy we employed exhibits lower missing rate, which ensures that the offloading mechanism does not introduce significant time overhead. In the experiments described in the main text, we chose a decay value of 0. I. Besides, to validate the effectiveness of our cache, we conducted an ablation study: running InfLLM without the GPU cache. The experimental results demonstrate that for encoding a 100K sequence, the addition of GPU cache reduces time costs from 21.5s 18.8s.
B Positional Encoding
In InfLLM, we assign all tokens beyond the local window size with the same positional encoding. Therefore, for the current tokens, we do not explicitly provide positional information for the context. But we think that the unidirectional nature of a decoder-only model allows to recognize the positional information of the context. For instance, assume sequence contains three spans SA, Sn. and Sc in order. When encoding Sc, although SA and SB are assigned the same positional encoding, the unidirectional nature of the decoder-only model allows the key-value hidden states of SA and Sg
16

---

inherently embeds their relative positional information: Sg can utilize information from SA during its encoding, while S can only access information from preceding parts of the sequence.
To verify the model's capability to capture the relative positional information of the context, we task with multiple pass keys for evaluation. In this task, each sequence
adopt the Retrieve. contains two pass keys, and the model is required to output these two pass keys in order. The data construction approach is consistent with that of oc-Bench (Zhang et al., 2023a), where the positions of the two pass keys are randomly selected. We created 50 sequences, each 64K in length. The experimental results reveal that in this task, InfLLM can output the values of the two pass keys in the correct order 100% of the time. This indicates that, although our positional encoding disregards the relative positional information of the context, the model can still effectively understand the context in sequence.
C External Experiments
C.1 Implementation Details
The context memory is constructed for all layers in LLMs. We set the size of our GPU cache 32, which is twice the number of loaded units for each step. We set the frequency score decay coefficient as 0.1. We adopt the half-float precision for all experiments. We use NVIDIA A100 or A800 to conduct our experiments. For the experiment that scales to 1, 024K context, we set the encoding chunk size as 2048, and the number of representative tokens as 1 to speed up experiments.
C.2 Performance on LongBench
We also employ LongBench Bai et al. (2023) the benchmark evaluate the effectiveness of InfLLM and baseline models. The evaluation results are shown in Table The results indicate that: (1) InfLLM outperforms other models capable of processing streaming inputs across various diverse tasks. It proves that the context information provided by the context memory can efficiently enhance the model performance. (2) When applying Llama-3 as the base model, both StreamingLLMM and LM-Infinite achieve only comparable or even worse performance than the original Llama-3. This indicates that while sliding window attention can effectively extend the context window size of LLMs, these models discard long-diss -iig-dnitncce contextual information, thereby failing to achieve effective long-sequence understanding. (3) Mistral can handle text lengths up to 32K, covering most instances in LongBench. In contrast, InfLLM, with window size of only 12K, achieves comparable or even superior performance on average. This further demonstrates InfLLM's s ability to filter out noise in long contexts, leading to better sengseeuennce understanding.
C.3 Experiments on Vicuna
In the previous sections, we demonstrated that InfLLM can extend the context windows of Llama-3 (with a maximum length of 8K) and Mistral (with maximum length of 32K) sev- eral hundred thousand tokens. To further vali- date the effectiveness of InfLLM, we apply to the Vicuna Chiang et al. (2023), which has maximum length of only 4K. The experimental results are shown in Table 6. The results show that we effectively extend Vicuna's is context length to 128K, achieving significant performance improvements on the Retrieve. Passkey and Cetrieve.Nuuber tasks. However, InfLLM can not show performance gains on the Retrieve. KV and Math.Find tasks. This is because the hidden vectors contained in Vieuna have limited ability to filter out noise extremely long texts, making it difficult for context memory to effectively locate relevant information in the more complex contexts of the Retrieve. and Math. tasks. In the future, It deserves further exploration to design more powerful memory mechanism.
Table 6: The results of Vicuna-based models.
R.PK
R.Num
R.KV
Math.F
Vicuna
5.08
4.41
1.40
11.71
InfLLM 99.15
81.69
0.60
11.14
17

---

Table 7:
The combination of InfLLM and models with continual pre-training, Yi-9B-200K (Yi-
200K).
Train-Free
R.PK
R.Num
R.KV
Choice
QA
Sum
Math.F
Yi-200K Xi-200K+InILLM
100.0
98.3
54.5
63.3
13.0
5.9
23,4
100.0
98.3
47.8
45.4
8,2
4.7
33.1
C.4 Combination of InfLLM with Yi-200K
We also present the results for the combination of InfL.LM and Yi-9B-200K (Young et al., 2024) in Table 7. From the results, we can observe that InfLLM can also achieve comparable results with Yi-9B-200K.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide detailed experimental results to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion about the limitation of this paper in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
We propose an empirical method without theoretical analysis and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide implementation details in Section 4.2 and Appendix C.I. Besides, provide the source code used in this paper.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We released our code for further research. The data used this paper is constructed by previous works, which haved cited in our papper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
public/guides/ Sotees//oooSSuuiiiinnnee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.ccppubiic//eec / ssi wwwiinssssniiiiiiiio sesionnyocy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in Section 4.1 and Appendix C.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The method proposed in this paper does not require any training. Therefore, there is no randomness in the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details in Appendix C.I and report the computation time consumption in Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http//eeurpp...eee cttppp..c..ppppeeeeeppp httpppulltltiiaptiiiitppp
Answer:
[Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss broader impact the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In this paper, we propose training-free free method to extend the context window of existing L.L.s.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cite the data, and code used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed readme the released code.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We donot require human annotation and evaluation in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "9gaCLD4Lku", "reviewer": "Reviewer_erR9", "summary": "This paper proposes breaking the context into blocks, and using a heuristic to compute representative tokens for the block. Then the attention score is first computed over these representative tokens, and the full attention is computed only over blocks corresponding to top-scoring representative tokens. The authors emphasize that the representative tokens are obtained without additional training. To improve memory usage, blocks are offloaded to CPU and only those selected for full attention are loaded into GPU. A cache is maintained to reduce offloading overhead.", "questions": "1. What is the overhead of offloading?", "limitations": "The authors mention avenues for future work. However, a disucssion around overheads and better/worse inference speed is missing.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "The paper is well written. The results are reported on multiple tasks and models.", "weaknesses": "The paper does not cover related work well. As a result there is a lack of novelty when considering existing work. Many ideas such as breaking the context into blocks, using a representative to decide usefulness of a block, and offloading to CPU are used and discussed in earlier work such as [1]. The idea of using a heuristic for obtaining the representative of block is also proposed in [2]. Though a different rule (max-pooling) is used in that work. No comparison with either of these works is included in the paper. In particular, while the authors emphasize the training-free nature of the proposed method, it is not clear how much performance is lost in comparison with a training-based method.\n\nGiven the above discussion, the main contributions of this paper are the specific formula used to obtain the representative tokens, and the low-level offloading implementation together with cache. However, these contributions are not fully highlighted and evaluated in the current version. 1) While the importance of using LRU for cache eviction over other baseline strategies such as random eviction is briefly studied in terms of cache misses, the efficiency of offloading is not discussed. In particular, it is not clear whether actual speedup is observed when using offloading. Interestingly the rate for cache misses is very low and it would be interesting to see the traffic between CPU and GPU in this scenario.  2) The formula is not compared with alternative methods such as [2].\n\n[1] Mohtashami A, Jaggi M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300. 2023 May 25.\n[2] Ren H, Dai H, Dai Z, Yang M, Leskovec J, Schuurmans D, Dai B. Combiner: Full attention transformer with sparse computation cost. Advances in Neural Information Processing Systems. 2021 Dec 6;34:22470-82.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our paper. Here are some clarifications regarding the points raised:\n\n### Q1: Comparison to Existing Memory-based Methods\nPlease refer to Q1, Q2, and Q3 in the global response. We are the first to construct a block-level training-free memory module for LLMs. Training-based methods require substantial computational resources and high-quality long-text pre-training and SFT data, which demand significant human effort. Moreover, in many real-world applications, such as ongoing conversations or LLM-driven agents, we need to process streaming input. In these scenarios, training-based methods are constrained by the maximum length used during their training process. Therefore, training-free context extrapolation is a crucial topic.\nExisting memory-based methods are mainly designed for pre-training language models from scratch. In contrast, InfLLM can be applied in a plug-and-play manner to all transformer-based methods, enhancing their ability to process long texts. Additionally, InfLLM is the first work to design a memory module specifically suited for training-free context extrapolation, implementing a comprehensive and efficient inference solution. Therefore, in this paper, we focus on the performance of memory-based methods in length extrapolation. In our revision, we will further discuss the differences between our work and existing memory-based methods.\n\nAs for the comparison with training-based methods, please refer to Table 2 in the original paper and Table 1 in the global response. We compared InfLLM with full-attention models, which can be considered the upper bound of long-context LLMs. The experimental results show that InfLLM can perform comparably to these models on InfiniteBench, demonstrating the effectiveness of InfLLM.\n\n### Q2: Efficiency in Offloading Mechanism\nAs stated in our paper, due to the semantic continuity of long text sequences, the memory units required for adjacent text spans are often highly similar. Therefore, utilizing a GPU cache can effectively reduce the communication between GPU and CPU. Even we employ the random strategy to manage GPU cache, the missing rate is quite low. Therefore, our findings are that the offloading mechanism should be widely-used for long-context inference, which will not incur much additional time costs.\n\nTo validate the effectiveness of our cache, we conducted an ablation study: running InfLLM without the GPU cache. The experimental results demonstrate that for encoding a 100K sequence, the addition of a GPU cache reduces our time costs from 21.5s to 18.8s.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U6HqNFCK8r", "reviewer": "Reviewer_5j15", "summary": "This paper focuses on the issue of extending context windows in LLMs by proposing a training-free block memory retrieval module. Specifically, aside from retaining initial tokens and local window tokens, other tokens are recalled using KNN at the block level. For efficient inference acceleration, LRU and chunk methods are employed to manage past key values in CPU or GPU memory. The proposed approach is tested on Mistral-32K and LLaMA-3-32K/1M models using benchmarks such as InfiniteBench and LongBench, with input tokens up to 214K. Results indicate that performance is generally maintained or improved across most tasks and models. However, there are notable performance drops in certain tasks when applying Long-context LLMs, such as the Choice task in Table 2.", "questions": "- **Q1**: Do you have any discussions or experimental results using other memory-based methods[1-3]?\n- **Q2**: Do you have results using other well-trained Long-context LLMs, like Yi-200K[4]? Could you explain the significant performance drop in the Choice task in Table 2?\n- **Q3**: Do you have any effective context windows benchmarks results, like Needle In A Haystack[5] or RULER[6]?\n- **Q4**: Do you have the ablation results for the block-level design? Could you explain why performance drops with decoding only?\n- **Q5**: Do you have the end-to-end latency results for different context windows?\n\n[4] https://huggingface.co/01-ai/Yi-34B-200K\n\n[5] https://github.com/gkamradt/LLMTest_NeedleInAHaystack\n\n[6] https://github.com/hsiehjackson/RULER", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The research problem analysis in this paper is significant and highly applicable.\n- The motivation behind the paper is sound. For each prompt request, relevant information is dynamic and sparse, which can be effectively handled through a KNN-based memory architecture.", "weaknesses": "1. The paper does not discuss the relationship and performance comparisons with other related memory-based methods[1-3], even though many require training.\n2. While InfLLM performs well on most LLMs without extended context windows, Table 2 shows performance drops in some tasks for Long-context LLMs with InfLLM.\n3. The datasets used in the experiments do not clearly reflect the changes in effective context window length when using InfLLM.\n4. There is a lack of necessary ablation experiments to demonstrate the effectiveness of the block-level design.\n5. The paper lacks results on end-to-end latency across different context windows.\n\n- [1] Unlimiformer: Long-Range Transformers with Unlimited Length Input, NeurIPS 2023.\n- [2] Focused Transformer: Contrastive Training for Context Scaling, NeurIPS 2023.\n- [3] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our paper. Here are some clarifications regarding the points raised:\n\n### Q1: Comparison to Existing Memory-based Methods\nPlease refer to Q1, Q2, and Q3 in the global response. Regarding the articles you mentioned, Unlimiformer employs token-level memory. From our experiments, token-level memory requires significant time for building the retrieval index. Even accelerated with Faiss, it still takes us 2 hours to process a 128K sequence. Focused Attention heavily relies on training unit representations, and the experimental results show that even with training, Focused Attention fails to achieve accurate retrieval on simple tasks like Passkey. Similarly, Infinite Attention uses linear attention mechanism to memorize distant tokens. However, this memory module suffers from catastrophic forgetting due to the limited memory state size, and experimental results demonstrate that this method cannot achieve general distant context memorization. Infinite Attention requires separate fine-tuning for different tasks, making it very impractical.\nIn summary, InfLLM can achieve efficient and general context memory, enhancing LLMs' long-text processing capabilities without requiring training.\n\n### Q2: Comparison to Well-trained Long LLM\nPlease refer to Q4 in the Global Response. InfLLM can still achieve comparable performance with Yi-9B-200K, demonstrating InsLLM's effectiveness. Regarding the performance drop in the Choice task, we believe this is due to the nature of the task, which requires retrieving four separate context sections while encoding a short span (the four options). InfLLM's retrieval during the pre-filling stage is conducted on a block basis, making it challenging to retrieve information for four semantically distinct options. Therefore, further research into how to efficiently improve the granularity of InfLLM's retrieval during the pre-filling stage is a crucial area for future investigation.\n\n### Q3: Evaluation on RULER\nPlease refer to Q5 in the Global Response.\n\n### Q4: Ablation Study for Block-level Design\nWe implement an LLM with token-level memories, and it takes 2 hours to process a sequence with 128K tokens. The time cost is unaffordable, and thus we can not present the results for token-level memories. As for performance drops in decoding only settings, it indicates that prefilling stage is quite important for long sequence processing. With only sparse attention in the prefilling stage, LLMs cannot capture the semantic similarity between distant tokens. Similar phenomenon is also observed in MInference.\n\n\n### Q5: Latency Across Different Context Windows\nIn our current implementation, during the memory lookup process, we need to compute the dot product between the query and all unit representations. Therefore, in terms of complexity, our current implementation still has O(n^2) complexity. Recent research has shown that different attention heads exhibit significant sparsity (MInference). We can potentially combine InfLLM with these findings to further reduce inference complexity.\nIn this paper, we emphasize InfLLM's context length extrapolation capability, which allows LLMs trained on shorter texts to better handle longer documents. Further reducing InfLLM's computational complexity is a direction for our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xoyzvgtVpI", "reviewer": "Reviewer_Srrj", "summary": "This paper introduces InfLLM, which is a training-free memory-based method for long context extension. The key mechanism is to incorporate the sliding window attention with an efficient context memory. Each token only attends to local and relevant contexts from the memory. The paper conducts extensive experiments to demonstrate the effectiveness.", "questions": "See the weakness section", "limitations": "The main limitation of this paper is that the author claims that the training-free long context extension method can reach a 1M context window, and is comparable to training-based methods. This is counter-intuitive, and based on my own experience, long context extension requires fine-tuning for real application usage. However, there is no theoretical proof or analysis to support this, and the empirical experiments are not sufficient.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow\n2. The proposed method can have good/comparable performance with other baselines on Infinite-bench and longbench, while being much efficient", "weaknesses": "1. Although the paper conducts many experiments on LongBench and InfiniteBench, the two benchmarks do not reflect long capabilities. I suggest that the paper should also include ruler[1] and long context code understanding benchmarks[2]. \n\n2. The author makes comparisons with the training-based long context extension method and RAG in Sections 4.4 and 4.5 respectively, concluding that they can achieve comparable results but with better efficiency. However, the baseline models chosen by the author are not strong enough. Thus, the results and conclusion are not convincing. For example, there are many high-quality long context LLMs on 128k context window, please add them as baselines to evaluate InfLLM's performance within 128k context window. \n\n3. The idea of using the initial token (e.g., streamingLLM) and selecting relevant tokens from past tokens for long context extension is incremental.\n\n[1] https://arxiv.org/abs/2404.06654\n\n[2] https://evalplus.github.io/repoqa.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our paper. Here are some clarifications regarding the points raised:\n### Q1: Evaluation on RULER\nPlease refer to Q5 in the Global Response.\n### Q2: Comparison to LLMs with 128K Context Window\nPlease refer to Q4 in the Global Response. We compare InfLLM with the full-attention model based on Yi-9B-200K. From the results, we can observe that InfLLM can still achieve satisfactory performance, which further proves the effectiveness of InfLLM.\n### Q3: Novelty\nPlease refer to Q1, Q2, and Q3 in the Global Response.\nWe argue that building a memory for LLMs is the simplest and most effective method to address long-context challenges. Previous algorithms have struggled to achieve optimal efficiency and effectiveness. Long-text processing is a highly application-oriented field, and therefore, implementing a simple idea with high effectiveness is more conducive to real-world deployment. We believe our empirical methods can greatly benefit LLM deployment for long sequences.\n\nThe simplicity of our approach, combined with its demonstrated effectiveness, makes it particularly valuable for practical applications. By focusing on a straightforward yet powerful memory mechanism, we have developed a solution that not only improves performance but also maintains feasibility for real-world implementation. This balance between simplicity and effectiveness is crucial in bridging the gap between theoretical advancements and practical deployment in the field of long-context language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZuPD3uVKn", "reviewer": "Reviewer_a1ZB", "summary": "This paper proposes a key/value retrieval-based method for long-context processing in Transformers. To address the distraction issue in long-context processing, the authors introduce an efficient method that retrieves blocks of relevant keys and values for attention computation. Through extensive evaluations on long-context processing benchmarks, the authors demonstrate the efficiency and effectiveness of the proposed method compared to existing state-of-the-art baselines.", "questions": "- Are the representative tokens different across layers? Can you observe any interesting patterns or insights regarding the representative tokens?\n- I am not convinced why a memory unit size of around 100 achieves the best performance in Figure 2-(c). (i.e., Wouldn't the factual/contextual information be contained in fewer tokens than that?) In the figure, the authors fix the total context size, so a smaller memory unit size retrieves more memory units. What happens when we fix the number of retrieved units and vary the size of each unit?", "limitations": "Yes, in Appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read.\n- The proposed method is very simple yet surprisingly effective.\n- The proposed method is training-free and can be applied to general Transformer models.\n- The authors provide extensive experimental results, demonstrating the effectiveness of the proposed approach for long-context processing.", "weaknesses": "- The paper does not introduce new technical challenges, methods, benchmarks, or phenomena. \n  - As I understand it, the proposed method adds a retrieval technique based on techniques from previous works (e.g., sink token, reassignment of positional encoding [1]). \n  - Additionally, the block-wise attention mechanism is not new (e.g., landmark attention [2]), and some methods like the kNN-Transformer use advanced retrieval techniques such as Faiss [3], which can cluster and construct blocks with similar elements. \n\nWhile I believe the paper is practically useful, the lack of novelty leads me to assign it a moderate score as a research paper.\n\n[1] \"Efficient Streaming Language Models with Attention Sinks\", ICLR 2024  \n[2] \"Landmark Attention: Random-Access Infinite Context Length for Transformers\", NeurIPS 2023  \n[3] \"Memorizing Transformers\", ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our paper. Here are some clarifications regarding the points raised:\n### Q1: Novelty\nPlease refer to Q1, Q2, Q3 in the Global Response. We implement an LLM with token-level memories, and it takes 2 hours to process a sequence with 128K tokens. The cost of time is unaffordable.\nWe argue that building a memory for LLMs is the simplest and most effective method to address long-context challenges. Previous algorithms have struggled to achieve optimal efficiency and effectiveness. Long-text processing is a highly application-oriented field, and therefore, implementing a simple idea with high effectiveness is more conducive to real-world deployment. We believe our empirical methods can greatly benefit LLM deployment for long sequences.\nThe simplicity of our approach, combined with its demonstrated effectiveness, makes it particularly valuable for practical applications. By focusing on a straightforward yet powerful memory mechanism, we have developed a solution that not only improves performance but also maintains feasibility for real-world implementation. This balance between simplicity and effectiveness is crucial in bridging the gap between theoretical advancements and practical deployment in the field of long-context language models.\n\n### Q2: Representative Tokens\nYes, different layers tend to choose different representative tokens. We conducted a case study on the selection of representative tokens, and from the results, we can broadly observe that in the top layers, the model generally selects words with stronger semantic information within spans. However, sometimes the model only selects common words, such as \"the\" or \"a\", as representations for spans. This indicates that there is still room for improvement in our unit representation approach.\n\n### Q3: Memory Unit Size\nThe segmentation of memory units is a trade-off between effectiveness and efficiency. When choosing a smaller memory unit size, it becomes easier to split a continuous statement into several semantically discontinuous spans, which can further interfere with retrieval effectiveness. Adopting a more dynamic span segmentation strategy is a future direction to improve InfLLM's performance. When we fix the number of retrieved units and vary the size of each unit, the results are as follows. As can be seen, when the unit size increases, the overall amount of relevant content retrieved also increases, leading to better performance.\n\n| Unit Size | R.KV  |\n| --------- | ----- |\n| 512       | 98.20 |\n| 256       | 96.00 |\n| 128       | 95.60 |\n| 64        | 93.20 |\n\nThese findings suggest that larger unit sizes tend to capture longer contexts, resulting in improved performance. However, this also highlights the importance of balancing unit size with computational efficiency and the risk of including irrelevant information. Future work could explore adaptive unit sizing strategies to optimize this trade-off for different types of content and tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Q1: Training-Free Context Extrapolation is Crucial
- Existing works for LLMs mainly focus on extending the model's context window through continued pre-training. However, these approaches face the following challenges:
   a) Long-sequence training requires substantial computational resources and large-scale high-quality datasets.
   b) The process of long-sequence training often leads to performance degradation on short-text data.
- In real-world applications, we envision LLMs capable of continuously processing unlimited streaming text input, such as in life-long personal assistants, LLM-driven agents. However, the training length of models will always be finite. Therefore, training-free context length extrapolation is crucial. Our method can handle streaming input, utilizing cost-effective CPU memory to store large key-value caches, allowing for scaling to longer texts.

### Q2: Training-Free Context Extrapolation is Challenging
- Effective block retrieval: As demonstrated in Focused Attention and Landmark Attention, differentiable block representation is critical for memory-based methods (Appendix C in Focused Attention, Section 4.2 in Landmark Attention). Our paper proposes a training-free block representation method that effectively mitigates the impact of irrelevant tokens on memory lookup.
- Efficient retrieval: Given the vast number of tokens in long contexts, token-level retrieval is prohibitively slow. Many existing training-based memory methods focus on constructing token-level memory units, such as unlimiformer and knn-LM, which usually require spending amounts of time for building retrieval index for large-scale tokens in each input sequence. It will lead to unaffordable time costs.
In summary, our paper focuses on the practical scenario of training-free context window extrapolation, proposing a memory module that requires no training and can effectively enhance the model's performance when processing streaming inputs. In the revision, we will add more discussion to clarify the contribution of our paper.

### Q3: Compared to Existing Memory-based Methods
As discussed in Q1 and Q2, training-free context extrapolation is crucial and challenging. 
- Existing memory-based methods are mainly proposed for **pre-training language models from scratch**, which requires amounts of computation resources and human efforts. 
- Besides, most existing memory-based methods focus on **token-level memory units** (KNN-Transformer, Unlimiformer, etc), which require a lot of time to build retrieval indexes for large-scale tokens in each input long sequence. We evaluate the time costs for token-level memory construction. For an LLM with token-level memories, it takes 2 hours to process a 128K sequence. Therefore, the cost of time is unaffordable.
- Some methods also adopt block-level memory (Landmark Attention, Focused Attention), these methods highlight the process of training effective block representations with long sequence data. From the evaluation on the Passkey retrieval task (Figure 1 in Focused Attention, Figure 3b in Landmark Attention), Landmark Attention can only achieve 80% accuracy for 32K sequences and Focused Attention can only achieve accuracy lower than 80% for 256K sequences.

### Q4: Compared to Well-Trained Long LLMs
In this study, we compared InfLLM with the continuously pre-trained Llama-3-8B-Instruct-Gradient-1048K. Furthermore, as suggested by Reviewer 5j15, we conducted a comparison combining InfLLM with Yi-9B-200K. The results are presented as follows. From the results, we can observe that InfLLM can also achieve comparable results with Yi-9B-200K.

|            | R.PK  | R.Num | R.KV | Choice | QA   | Sum  | Math.F | Avg. |
| ---------- | ----- | ----- | ---- | ------ | ---- | ---- | ------ | ---- |
| Yi-9B-200K | 100.0 | 98.3  | 54.5 | 63.3   | 13.0 | 5.9  | 23.4   | 51.2 |
| InfLLM     | 100.0 | 98.3  | 47.8 | 45.4   | 8.2  | 4.7  | 33.1   | 48.2 |

### Q5: Evaluation on RULER
LongBench and InfiniteBench are two widely used long-text evaluation datasets, encompassing tasks such as context retrieval, question answering, and summarization. To further demonstrate the effectiveness of InfLLM across various tasks, we conducted additional evaluations using the RULER benchmark with 128K tokens. The results are shown in the following Table. From the results, we can observe that InfLLM can still outperform the sliding window attention mechanism.

|        | NIAH-Single | NIAH-MQ | NIAH-MV | NIAH-MK | FreqWord | Var-Track | QA   | Commondword | Avg. |
| ------ | ----------- | ------- | ------- | ------- | -------- | --------- | ---- | ----------- | ---- |
| Stream | 5.3         | 4.9     | 5.3     | 3.5     | 90.1     | 10.0      | 20.8 | 0.1         | 17.5 |
| InfLLM | 43.2        | 8.3     | 9.0     | 5.3     | 89.8     | 15.1      | 19.6 | 0.1         | 23.8 |

Meta-review: This paper proposes a new training-free context length extrapolation method for LLMs. 

Intially, this paper had three marginal acceptance and one rejection scores. 

By the rebuttal, one reviwer raised his/her score to 7, while other reviewrs kept their score.

For clear decision, AC carefully focused on the discussion between the authors and R-erR9.

It seems that both viewpoints are valuable and reasonable. In practical, in the viewpoint of LLM practices in real-world, fine-tuning-based context explortation apporaches are practical without requiring too heavy GPU resources. 
However, academic research needs to explore different approach beyond practice for more fundamental solutions to make a breakthrough.

Considering the contributions of this paper and this perspective, this paper seems to satisfying the quality for NeurIPS.

So, AC recommends accepting this paper. 
AC asks the augthors to address R-erRP's viewpoints properly in the cam-ready version.
Decision: Accept (poster)