InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

Authors: chaojun  xiao
pengle  zhang
xu  han
guangxuan  xiao
yankai  lin
zhengyan  zhang
zhiyuan  liu
maosong  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:02:11

Abstract: Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to 1,024K, InfLLM still effectively captures long-distance dependencies. Our code can be found at https://github.com/thunlp/InfLLM.

Main paper: 
 introduction: 
 Recently, large language models (LLMs) have achieved profound accomplishments in various tasks (Brown et al., 2020; Bommasani et al., 2021; Han et al., 2021; Touvron et al., 2023; Meta, 2024) . Their ability to follow complex instructions shed light on the realization of artificial general intelligence (OpenAI, 2023; Ouyang et al., 2022) . With the blooming of LLM-driven applications, such as agent construction (Park et al., 2023; Qian et al., 2023; Wang et al., 2024a) and embodied robotics (Driess et al., 2023; Liang et al., 2023) , enhancing the capability of LLMs to process streaming long sequences become increasingly crucial. For instance, LLM-driven agents are required to process information continuously received from external environments based on all their historical memories, necessitating a robust capability for handling long streaming sequences.Due to limitations caused by unseen lengthy inputs (Han et al., 2023) and distracting noisy contexts (Liu et al., 2023; Tworkowski et al., 2023) , most LLMs, pre-trained on sequences consisting of only a few thousand tokens, cannot process longer sequences (Press et al., 2022; Zhao et al., 2023) .Common solutions usually involve continually training LLMs on longer sequences but further result in substantial costs and require large-scale high-quality long-sequence datasets (Xiong et al., 2023; Li et al., 2023) . And the continual training process on longer sequences may weaken the performance of LLMs on short contexts (Ding et al., 2024) . In view of this, improving the length generalizability of LLMs without further training receives extensive attention, trying to make LLMs trained on short sequences directly applicable to long sequences.In this paper, we propose a training-free memory-based approach, named InfLLM, for streamingly processing extremely long sequences with limited computational costs. Specifically, InfLLM incorporate the sliding window attention (Xiao et al., 2023; Han et al., 2023) with an efficient context memory, where each token only attends to local contexts and relevant contexts from the memory. Considering the sparsity of attention score matrices, processing each token typically requires only a small portion of its contexts (Zhang et al., 2023b) , and the remaining irrelevant contexts act as noise, leading to attention distraction issues (Tworkowski et al., 2023) . We thus construct an external memory containing distant context information. Only relevant information within the memory is selected for each computation step, and other irrelevant noises are ignored. Owing to this, LLMs can understand whole long sequences using a finite-size window and avoid noisy contexts.The vast amount of noisy context tokens in long sequences poses significant challenges to effective and efficient memory lookup. To address these challenges, we design a block-level context memory mechanism. Specifically, InfLLM organizes past key-value vectors into blocks, each containing a continuous token sequence. Within each block, the semantically most significant tokens that receive the highest attention scores are selected as the unit representation for subsequent relevance computation in memory lookup. This design offers two primary benefits: (1) Effective Lookup: The coherent semantics of each block can more effectively fulfill the requirements for relevant information retrieval compared to single tokens. The selection of unit representations minimizes the interference of unimportant tokens in relevance computation, enhancing the overall hit rate of memory lookup. (2) Efficient Lookup: The block-level memory unit eliminates the need for per-token relevance computation, significantly reducing computational costs. Moreover, block-level units ensure contiguous memory access, thus minimizing memory loading costs and enhancing computational efficiency. Furthermore, considering the infrequent usage of most units, InfLLM offloads all units on CPU memory and dynamically retains the frequently used units on GPU memory, significantly reducing GPU memory usage. Notably, the block-level memory mechanism in InfLLM does not involve any additional training, and can be directly applied to any LLMs.To evaluate the effectiveness of InfLLM, we employ Mistral-7B-inst-v0.2 (Jiang et al., 2023) and Llama-3-8B-Instruct (Meta, 2024) as base models, which are pre-trained on the sequences containing no more than 32K and 8K tokens. We use two widely-used benchmarks, ∞-Bench (Zhang et al., 2023a) and Longbench (Bai et al., 2023) , for evaluation. Especially, the average sequence length in ∞-Bench exceeds 100K tokens, which is challenging for most existing LLMs. Compared to typical methods that continually train LLMs on longer sequences, the experimental results demonstrate that InfLLM enables the LLMs pre-trained on the sequences containing a few thousand tokens to achieve comparable performance without any additional training. Moreover, we examine InfLLM on the sequences containing 1, 024K tokens, and InfLLM can still effectively capture long-distance dependencies, demonstrating the potential of InfLLM in scenarios involving long streaming inputs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We provide detailed experimental results to support our claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Enabling LLMs to process long sequences has been extensively studied (Dong et al., 2023; Tay et al., 2023; Huang et al., 2023) and can generally be categorized into two main approaches: context length extrapolation and efficient context computation. The former aims to enable LLMs trained on short sequences to process much longer sequences. The latter focuses on enhancing the computational efficiency of attention layers, allowing efficient pre-training LLMs from scratch to process longer sequences. Although the focus of this paper is context length extrapolation, we also detailedly introduce efficient context computation. We also present the relevant works for memory-based models.Context Length Extrapolation. Due to the high computational and memory requirements, the training of LLMs is often restricted to short sequences. Directly applying LLMs to long sequences will suffer from out-of-domain and distraction challenges caused by lengthy and noisy inputs (Han et al., 2023; Tworkowski et al., 2023) . Consequently, context length extrapolation has garnered attention as a method to improve the sequence length for LLMs without incurring additional training. The earliest approaches involve designing new relative positional encoding mechanisms during pre-training (Press et al., 2022; Sun et al., 2023) . Subsequent studies mainly focus on the widely-used rotary position embedding (RoPE) (Su et al., 2021) , and propose to achieve length extrapolation by downscaling or reusing the original position indices (Chen et al., 2023b; Peng et al., 2023; Chen et al., 2023a; Jin et al., 2024; An et al., 2024) . These works can alleviate the out-of-domain issue from the unseen length, but can not alleviate the distraction challenge of noisy contexts. To address this, Xiao et al. (2023) and Han et al. (2023) employ the sliding window attention mechanism and directly discard all distant contexts to streamingly read extremely long sequences. However, as these models overlook information from distant tokens, they can not capture the long-distance dependencies for long-text understanding. In this paper, InfLLM utilizes the sliding window attention mechanism, and additionally constructs an efficient context memory to provide LLMs with relevant context information, enabling LLMs to effectively read and understand extremely long sequences.Efficient Context Computation. The quadratic computational complexity of the attention layers is a primary factor limiting the lengthy sequence-processing capabilities of LLMs. Thus, numerous scholars have endeavored to design efficient attention mechanisms, including the utilization of sparse attention (Zaheer et al., 2020; Beltagy et al., 2020; Child et al., 2019; Ainslie et al., 2020; Zhao et al., 2019) , approximating attention computations using kernel functions (Kitaev et al., 2020; Wang et al., 2020; Katharopoulos et al., 2020) , and replacing the attention layer with linear-complexity state-space models (Gu et al., 2022; Gu & Dao, 2023) . These approaches necessitate a modification in the model architecture, requiring retraining the models. Simultaneously, many researchers enhance the inference efficiency by evicting useless key-value vectors to reduce computation (Zhang et al., 2023b; Li et al., 2024; Ge et al., 2023) . These methods can not extrapolate the context window of LLMs without further training due to out-of-domain issues caused by unseen positions. Recently, some researchers begin to explore the intrinsic sparse attention patterns of long-context LLMs and discard the redundant attention computation for acceleration (Jiang et al., 2024) .Memory-based Models. Memory networks have been studied for decades, which are proven effective in providing models with additional knowledge and information storage capabilities (Graves et al., 2014; Weston et al., 2015; Sukhbaatar et al., 2015; Miller et al., 2016) . With the success of pretrained models, memory layers have also been gradually applied in the training processes of recurrent transformer layers, enabling models to process long sequences recursively (Dai et al., 2019; Rae et al., 2020; Khandelwal et al., 2020; Wu et al., 2022; Bertsch et al., 2023; Munkhdalai et al., 2024) . These works split sequences into segments, encoding each segment individually, and use memory to store context information from preceding segments. While these approaches are similar in concept to InfLLM, they involve modifications to the model architecture and requires further training the whole model. Besides, most existing memory-based methods focus on token-level memory units (Wu et al., 2022; Bertsch et al., 2023) , which require a lot of time to build retrieval indexes for large-scale tokens in each input long sequence. Some methods also adopt block-level memory (Mohtashami & Jaggi, 2023; Tworkowski et al., 2023) , these methods highlight the process of training effective block representations with long sequence data. In contrast, we aim to explore the inherent characteristics of LLMs, and propose a training-free memory module for long-text understanding.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We provide discussion about the limitation of this paper in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 As shown in Figure 1 , InfLLM builds a training-free context memory to efficiently provide highlyrelevant contexts for each token, endowing the sliding window attention mechanism with the ability to capture long-distance dependencies.The main restrictions for improving the length generalizability of LLMs come from the out-of-domain and distraction issues caused by the lengthy and noisy contexts. To address these, following previous works (Xiao et al., 2023; Han et al., 2023) , we adopt the sliding window attention mechanism, which only considers local tokens for each step. Additionally, we construct an extra context memory module to provide relevant context information to capture long-distance dependencies.Figure 1 : The illustration of InfLLM. Here, the current tokens refer to tokens that need to be encoded in the current computation step. The past key-value vectors can be divided into the initial tokens, evicted tokens, and local tokens, arranged the furthest to the nearest relative to the current tokens. For each computation step, the context window consists of the initial tokens, relevant memory units, and local tokens. Specifically, we denote the long input sequence as s = {t i } l i=1 . Due to the limited GPU memory, instead of encoding the whole s at once, we encode the input sequence s chunk-by-chunk and generate the output token-by-token. For each computation step, the inputs consist of past key-value vectors P = {(k j , v j )} l P j=1 and current tokens X = {t i+l P } l X i=1 . For encoding steps, l X equals the chunk size, and for decoding steps, l X equals one.According to the distances from current tokens, we can divide P into three groups: initial tokens, I = P [1:l I ] , evicted tokens, E = P [l I +1:l P -l L ] , and local tokens, L = P [l P -l L +1:l P ] , arranged from the furthest to the nearest relative to the current tokens. Here, l P , l I , l L refer to the length of past key-value vectors, initial tokens, and the local window size. All evicted tokens, E, are stored in the context memory, consisting of multiple memory units. For each step, InfLLM concatenates the initial tokens, relevant memories units from context memory, and local tokens to form the current key-value cache, C = Concat(I, f (X, E), L). f (•) refers to the lookup operation of context memory. The attention output is calculated as:O = Attn [QX, Concat(C k , KX), Concat(C v , VX)] .Here, Q, K, and V are parameters in attention layers, C k and C v refer to the key and value vectors in C. If f (•) always returns empty sets, InfLLM is degenerated into LM-Infinite (Han et al., 2023) and Streaming-LLM (Xiao et al., 2023) , which directly discards distant contexts.Previous findings indicate that the attention score matrices of LLMs are sparse, and we can generate the same outputs with only a small portion of key-value vectors preserved (Zhang et al., 2023b) . Inspired by this, we design a context memory to efficiently look up relevant contexts from large-scale evicted tokens and ignore irrelevant ones to save computational costs. The most intuitive way is to construct a memory consisting of token-level memory units for every past key-value vectors, and every attention head separately, which would result in massive memory units, unacceptable computation, and non-contiguous memory access costs. Thus, considering the local semantic coherence of long sequences, we split the past key-value vectors into blocks, each serving as a memory unit, and conduct memory lookup at the block level to reduce the costs while preserving the performance.In this subsection, we will introduce the details of the block-level memory units. Then we present the method to assign positional embeddings for selected relevant memory units and cache management for the context memory.Block-Level Memory Units. Block-level memory units can save computation costs compared to token-level ones. It also poses new challenges for unit representations, which are supposed to contain the semantics of the entire unit for effective relevance score computation and be memory-efficient for context length scalability. Traditional methods usually involve training an additional encoder to project a given unit into a low-dimension vector. Inspired by the token redundancy in hidden states (Goyal et al., 2020; Dai et al., 2020) , we select several representative tokens from the entail blocks as the unit representation. For the m-th token, we define the representative score as:r m = 1 l L l L j=1 q m+j • k m ,where q m+j is the query vector for (m + j)-th token and k m is the key vector m-th token. Intuitively, r m represents the significance of the m-th token in its corresponding local window, indicating the extent of its influence on other tokens within the local window. The computation of representative scores requires no additional parameters.Formally, given the evicted tokens, E, we split it into several memory units, each containing l bs tokens.For each unit, the r k tokens with the highest representative scores are selected as representative tokens.Generally, r k is a small positive integer. Let us denote a memory unit asB = {(k B j , v B j )} l bs j=1, and the representative tokens of this unit asR(B) = {(k B bj , v B bj )} r k j=1 .For the memory lookup phrase, only k m units with the highest relevance scores are loaded for the current attention computation. We calculate the relevance score between B and current tokens X as:sim(X, B) = l X i=1 r k j=1 q i+l P • k B bj .Notably, the representative tokens selection is a training-free method to obtain the unit representations.Here, we can also train an additional encoder to generate more expressive unit representations, which we leave for future work.Positional Encoding. Existing LLM training usually employs a finite number of positional encodings, which encounter out-of-domain distribution challenges when directly applied to longer sequence processing (Han et al., 2023) . Besides, in InfLLM, the current key-value cache is composed of some discontinuous text blocks, and directly assigning continuous positional encodings to them would also lead to mismatch issues and confuse the model. Therefore, inspired by previous works (Raffel et al., 2020; Su, 2023) , we assign all tokens beyond the local window size with the same positional encodings. Specifically, the distance between tokens in context memory units and current tokens is set as l L .Cache Management. To enable LLMs to process extremely long sequence streams while capturing the semantic relevance contained in the long contexts, we need to retain all memory units and look up them at each computation step. Considering the infrequent usage of most units, we employ an offloading mechanism, storing most memory units in CPU memory and only preserving the representative tokens and memory units needed in current steps in GPU memory. Additionally, given the semantic coherence of long sequences, where adjacent tokens often require similar memory units, we allocate a cache space in GPU memory, managed using a least recently used strategy. This approach allows for efficient encoding of extremely long sequences using limited GPU memory.From the observation, our offloading mechanism enables InfLLM to process sequences consisting of 100K tokens with only 26G VRAM. Besides, the miss rate of our GPU cache is quite low, which means the offloading mechanism does not introduce significant time overhead in memory loading while saving GPU memory usage. The details can be found in the Appendix.Furthermore, for extremely long sequences, the representative tokens of each unit can also be offloaded to the CPU memory, constructing an efficient k-nearest-neighbor index, and thereby further reducing computational complexity.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: We conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
settings: 
 Datasets. We adopt representative tasks in a widely-used long document benchmark, ∞-Bench (Zhang et al., 2023a) for evaluation. We adopt the English datasets for evaluation as the base models are mainly pre-trained on English corpus. The datasets in ∞-Bench cover diverse tasks including question answering, summarization, context retrieval, and mathematic computing. The average length for ∞-Bench is 145.1K. The 95% quantile for sequence lengths is 214K, which is far beyond the maximum length of the base models. Detailed statistics and task descriptions of these datasets are listed in the Appendix. Besides, we also conduct an evaluation on LongBench (Bai et al., 2023) . The results for LongBench can be found in the Appendix.Baseline Models. To verify the effectiveness of our proposed method, we compare InfLLM with the following competitive baseline models:(1) Original models: we present the performance of the original LLMs without context length extrapolation.(2) Position downscaling and resuing: NTKaware scaled RoPE (NTK) (LocalLLaMA, 2023) designs a nonlinear interpolation method, which basically changes the rotation base of RoPE. SelfExtend reuse the position ids across neighboring tokens, which makes the extended relative positions in the scope of the training context window.(3) Sliding window: these methods apply the sliding window mechanism to discard distant contexts, including LM-Infinite (Infinite) (Han et al., 2023) and StreamingLLM (Stream) (Xiao et al., 2023) . Therefore, for each attention computation step, the input length does not exceed the context window.(5) Key-value eviction: KV eviction methods aim to discard useless key-value vectors during long sequence processing and thus are usually used to reduce the computation complexity. We present the results of a widely-used key-value eviction method, H2O (Zhang et al., 2023b) . The key-value eviction method cannot generalize to longer sequences due to the unseen position embeddings and is expected to achieve unsatisfactory performance.Here, InfLLM and the models with the sliding window mechanism can be used to process extremely long streaming inputs. For NTK and SelfExtend, we extend the context window to 128K, which enables LLMs to process most instances in ∞-Bench.In this paper, we aim to enable LLMs trained with limited sequence length to read and understand extremely long sequences without further training. We adopt Mistral-7B-Instruct-v0.2 (Jiang et al., 2023) and Llama-3-8B-Instruct (Meta, 2024) as our base models. The maximum length of Mistral-7B-Instruct-v0.2 and Llama-3-8B-Instruct is 32K and 8K, respectively.For our model, we set the encoding chunk size as 512, and the memory unit size for past key-value vectors, l bs , as 128. The number of representative tokens, r k , is set as 4. For both Mistral-based and Llama-3-based InfLLM, we set the local window size as 4K. For Mistral-based InfLLM, we load 96 relevant memory units for each step, and for Llama-3-based InfLLM, we load 32 relevant memory units. The number of initial tokens is set as 128 for LM-Infinite, StreamingLLM, and InfLLM to cover the system prompts and task descriptions. We adopt FlashAttention (Dao, 2023) to accelerate experiments for all baseline models. Please refer to the Appendix for more details.The results for Mistral-based models and Llama-3-based models are reported in Table 1 . From the results, we can observe that: (1) Compared to models with the sliding window mechanism, which can also read extremely long sequences, our method demonstrates a significant performance improvement. This indicates that the context memory in InfLLM can accurately supplement LLMs with relevant contextual information, enabling efficient and effective understanding and reasoning on long sequences.(2) The position downscaling and resuing methods, NTK and SelfExtend, tend to compromise model performance while extending the sequence length to 128K. That is because these models cannot address the distraction issue caused by noisy contexts. In contrast, our model can consistently enhance performance for extremely long sequences. We successfully generalize Llama-3 from a 8K length to more than 16 times its length, achieving commendable performance on the ∞-Bench.(3) The position downscaling and resuing methods can increase the maximum sequence length of LLMs but also raise the computational and memory costs, limiting these methods' application. In contrast, InfLLM utilizes block-level memory and offloading mechanism, enabling efficient processing of long sequences within limited resources.In this paper, we focus on expanding the context window of LLMs without additional training. In this section, we compare InfLLM with models that undergo continual training on long sequences in terms of both performance and efficiency. Specifically, we select Llama-3-8B-Instruct-Gradient-1048k (Llama-1M)3 , which have been further fine-tuned on long-text data and chat datasets, extending its context window to 1048K. Besides, we also employ InfLLM on the Llama-1M, where we set the local window as 4K and selected memory size as 4K. We present the results on ∞-Bench, the GPU memory usage, and time consumption in Table 2 . From the results, we can observe that: (1) Compared to models that have undergone continual training on long sequences, InfLLM can achieve comparable or even superior results without any additional training. This suggests that LLMs inherently possess the capability to identify key information in long sequences and to understand and reason effectively. Notably, Llama-1M requires 512 GPUs for continual training, which is unaffordable for many researchers. In contrast, InfLLM does not require any training, which indicates the practicability of InfLLM.(2) In terms of efficiency, InfLLM achieves a 34% decrease in time consumption while using only 34% of the GPU memory compared to the full-attention models. Moreover, at longer sequence lengths of 256K tokens, the full-attention baseline fails due to out-of-memory errors, while InfLLM can efficiently process sequences up to 1024K tokens on a single GPU.(3) InfLLM can also be directly combined with the model with continual training and achieve comparable or even superior results with only 8K context window. It indicates that InfLLM can also serve as an efficient way to improve the inference speed.InfLLM leverages the intrinsic capacity of LLMs to construct a context memory for gathering token-relevant information, a concept similar to retrieval augmented generation (RAG) (Lewis et al., 2020; Nakano et al., 2021) . However, compared to using RAG, where historical contexts are treated as a searchable database for long-sequence understanding (Xu et al., 2023) , InfLLM has several advantages: (1) Training-Free: RAG requires additional retrieval data to train a retrieval model, whereas InfLLM is training-free and applicable to any LLMs. Besides, RAG also necessitates finetuning LLMs to adapt to the inputs augmented by the retrieved knowledge.(2) Broader Applicability: RAG models are usually limited by the performance of their retrieval components. Besides, existing retrieval models will suffer from out-of-distribution issues, struggling to perform well on tasks outside their training distribution (Lin et al., 2023; Muennighoff et al., 2023) . This limitation adversely affects the overall performance of the RAG system. In contrast, InfLLM has no specific requirements for tasks and can be feasibly used for long sequences. To verify the generalization capabilities of InfLLM, we conduct experiments to comparing RAG and InfLLM on three context retrieval tasks. We utilize E5-mistral-7B-instruct (Wang et al., 2024b) as the retrieval model. The results are shown in Table 3. Our findings demonstrate that even without additional data or training, InfLLM can consistently outperform RAG models, underscoring its superior generalization capabilities. The dependency on an external retrieval model makes RAG less flexible in handling diverse tasks.InfLLM relies on the context memory to look up relevant information. We further explore the impact of core components in the context memory, specifically the representative tokens and memory units.The results are shown in Figure 2 .Different Number of Representative Tokens. InfLLM splits key-value vectors into memory units and selects several representative tokens from the unit to serve as the unit representations. Consequently, the ability of these representative tokens to semantically represent the entire unit directly impacts the model's performance. We conduct experiments with the number of representative tokens as {1, 2, 4, 8}. The results are shown in Figure 2a . It is observed that as the number of representative tokens increases, there is a trend of improvement in the model performance, which indicates that more representative tokens tend to better represent the semantic content of the memory units. However, it is noted that when the number of representative tokens reaches 8, there is a slight performance decrease. This decline can be attributed to the inclusion of semantically irrelevant tokens as unit representations. More efficient and powerful unit representations will further enhance model performance for future work.Different Number of Selected Units. The selected units are utilized to provide relevant context to LLMs. We conduct experiments with the number of units set as {2, 4, 8, 16, 32, 64, 96, 128}. From Figire 2b, we can observe that as the number of selected units increases from 1 to 32, the model performance significantly improves, which is attributed to that more units imply a greater recall rate of relevant content. Larger unit quantity also leads to an increase in the required memory scheduling time and the computational time for attention. Therefore, further enhancing lookup accuracy remains a crucial direction for improving the efficiency of InfLLM.Different Memory Unit Size. Each memory unit is supposed to be a coherent semantic unit. Excessively large unit sizes can hinder precise lookup, while a small size will increase the computational overhead of memory lookup. We evaluate InfLLM with the unit size as {32, 64, 128, 256} and keep the total context length as 12K. The results are shown in Figure 2c . It can be observed that the optimal unit size varies for different tasks due to the varying characteristics of input sequences. For example, in Retrieve.KV, a key-value pair constitutes a semantic unit, while in Math.Find, a single number represents a semantic unit. Employing heuristic rules to segment context can easily lead to suboptimal performance. Therefore, exploring how to dynamically segment context is an important direction for future research.To further verify the effectiveness of dynamic memory lookup and unit representations, we conduct ablation studies in this section. The results are shown in Table 4 . Unit Representation. We design a block-level memory for efficient context information lookup.We select several representative tokens as the unit representations for relevance computation. We present the results of InfLLM with another training-free representation method (Mean Repr), which computes the representation by averaging the key vectors in a memory unit. From the results, we can observe that InfLLM with average representations can also present competitive performance. It indicates that the original attention vectors in LLMs are effective for relevance score computation, and exploring more efficient unit representations is an important future direction. Figure 3 : The results on sequences with different lengths.To assess the effectiveness of InfLLM on extremely long sequences, in this subsection, we scale the sequence length to 1024K to evaluate the capacity of InfLLM to capture contextual relevance in long sequences. Specifically, we adopt the Retrieve.PassKey task in ∞-Bench for evaluation. This task prompts LLMs to find a 5-digit sequence among lengthy and noisy contexts, which requires LLMs to locate relevant information among long sequences effectively. We automatically generate inputs with {32, 64, 128, 256, 512, 768, 1024} thousand tokens and for each length, we generate 50 instances for evaluation. We adopt Mistral as the base model.The results are shown in Figure 3 . From the results, we can observe that InfLLM can accurately locate the key information from length noises and achieve 100% accuracy even when the context length scales to 1024 thousand tokens. However, LM-Infinite can only attend to the tokens within the local window, which leads to a rapid decline in its performance as the sequence length increases. It proves that InfLLM can accurately capture the long-distance dependencies for effective long-sequence reasoning. 
conclusion: 
 In this paper, we propose a training-free method to improve the length generalizability of LLMs.Based on the sliding window attention mechanism, we construct an additional context memory module, which can help LLMs select relevant information from massive contexts to capture longdistance dependencies. The experiments on two widely-used long-text benchmarks show that InfLLM can effectively improve the ability of LLMs, which are trained on sequences with a few thousand tokens, to process extremely long sequences. In the future, we will explore efficient training of the context memory module to further enhance the model performance. Besides, combining the key-value cache compression methods with InfLLM can further reduce the computational and memory costs.We hope InfLLM can boost the development of streaming applications of LLMs. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impact in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: In this paper, we propose a training-free method to extend the context window of existing LLMs.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite the data, and code used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: This paper presents work whose goal is to advance the field of long sequence processing for large language models. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.In this paper, we propose InfLLM, a method for extending the context window of LLMs without additional training. We verify the effectiveness of our model using a widely-used long-text evaluation benchmark ∞-Bench. However, our method still has the following limitations: (1) We store a large amount of past key-value (KV) cache in the CPU memory, which increases CPU memory usage.In the future, we can reduce CPU memory requirements by integrating techniques like KV cache quantization.(2) While InfLLM reduces the computational overhead for processing long texts in LLMs, there is still room for speed-up. In the future, we can further enhance the inference speed of InfLLM by integrating it with inference frameworks like llama.cpp4 and vllm (Kwon et al., 2023) .A Cache Management Strategy Due to the massive amount of memory units for extremely long sequences, we adopt an offloading mechanism to save GPU memory costs. Considering the infrequent usage of memory units, we offload most memory units to CPU memory and only preserve the frequently used memory units and current needed memory units in the GPU memory. To this end, we maintain a cache in GPU memory to effectively utilize GPU memory and reduce the communication between CPU and GPU. The size for our GPU cache is fixed, and therefore we design a least recently used (LRU) strategy for cache management. In this section, we will introduce the management strategy in detail.Loading Memory Units For each computation step, we first compute the relevance scores for each memory unit to determine which units should be used. Then, for each needed memory unit, we first search it in our cache. If there is no hit, then we proceed with the transfer from CPU memory to GPU memory.Offloading Memory Units After the attention computation, we need to offload redundant memory units to keep the GPU cache fixed. To this end, we apply an LRU strategy. Specifically, for each memory unit loaded into our GPU cache, we assign a frequency score s b for it, which will be used to determine whether this unit should be maintained in the GPU cache or offloaded to CPU memory to save GPU memory costs. The frequency scores are updated after the attention computation. Specifically, we update the score as follows:EQUATIONwhere l u represents the number of current tokens involved in this lookup, attention_score(q, k) denotes the attention score between Q with respect to k (ranging from 0 to 1) obtained after performing the attention computation. d is a hyper-parameter, representing the decay coefficient, used to incorporate the influence of previous lookups. After each attention computation, we sort all the memory units in our GPU cache according to their frequency scores s b , and offload the units with the lowest scores back to the CPU memory.To verify the effectiveness of our cache management strategy, we evaluate the cache missing rate of different cache management strategies on a sample of data from the GovReport dataset. Specifically, we compare our LRU strategy with (1) Random: randomly selecting units from the GPU cache to (2) First-in-first-out (FIFO): offload the unit that is first loaded in the GPU cache. The results are illustrated in Figure 4 . It is observable that the LRU strategy we employed exhibits a lower missing rate, which ensures that the offloading mechanism does not introduce significant time overhead. In the experiments described in the main text, we chose a decay value of 0.1. Besides, to validate the effectiveness of our cache, we conducted an ablation study: running InfLLM without the GPU cache. The experimental results demonstrate that for encoding a 100K sequence, the addition of a GPU cache reduces our time costs from 21.5s to 18.8s.In InfLLM, we assign all tokens beyond the local window size with the same positional encoding. Therefore, for the current tokens, we do not explicitly provide positional information for the context. But we think that the unidirectional nature of a decoder-only model allows it to recognize the positional information of the context. For instance, assume a sequence contains three spans S A , S B , and S C in order. When encoding S C , although S A and S B are assigned the same positional encoding, the unidirectional nature of the decoder-only model allows the key-value hidden states of S A and S B We also present the results for the combination of InfLLM and Yi-9B-200K (Young et al., 2024) in Table 7 . From the results, we can observe that InfLLM can also achieve comparable results with Yi-9B-200K.NeurIPS Paper Checklisthttps://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048khttps://github.com/ggerganov/llama.cpp 


Decision: This paper proposes a new training-free context length extrapolation method for LLMs. 

Intially, this paper had three marginal acceptance and one rejection scores. 

By the rebuttal, one reviwer raised his/her score to 7, while other reviewrs kept their score.

For clear decision, AC carefully focused on the discussion between the authors and R-erR9.

It seems that both viewpoints are valuable and reasonable. In practical, in the viewpoint of LLM practices in real-world, fine-tuning-based context explortation apporaches are practical without requiring too heavy GPU resources. 
However, academic research needs to explore different approach beyond practice for more fundamental solutions to make a breakthrough.

Considering the contributions of this paper and this perspective, this paper seems to satisfying the quality for NeurIPS.

So, AC recommends accepting this paper. 
AC asks the augthors to address R-erRP's viewpoints properly in the cam-ready version.

Reviews: review: paper_summary: paper_summary: This paper proposes breaking the context into blocks, and using a heuristic to compute representative tokens for the block. Then the attention score is first computed over these representative tokens, and the full attention is computed only over blocks corresponding to top-scoring representative tokens. The authors emphasize that the representative tokens are obtained without additional training. To improve memory usage, blocks are offloaded to CPU and only those selected for full attention are loaded into GPU. A cache is maintained to reduce offloading overhead.
strength_weakness: strength_weakness: The paper does not cover related work well. As a result there is a lack of novelty when considering existing work. Many ideas such as breaking the context into blocks, using a representative to decide usefulness of a block, and offloading to CPU are used and discussed in earlier work such as [1]. The idea of using a heuristic for obtaining the representative of block is also proposed in [2]. Though a different rule (max-pooling) is used in that work. No comparison with either of these works is included in the paper. In particular, while the authors emphasize the training-free nature of the proposed method, it is not clear how much performance is lost in comparison with a training-based method.

Given the above discussion, the main contributions of this paper are the specific formula used to obtain the representative tokens, and the low-level offloading implementation together with cache. However, these contributions are not fully highlighted and evaluated in the current version. 1) While the importance of using LRU for cache eviction over other baseline strategies such as random eviction is briefly studied in terms of cache misses, the efficiency of offloading is not discussed. In particular, it is not clear whether actual speedup is observed when using offloading. Interestingly the rate for cache misses is very low and it would be interesting to see the traffic between CPU and GPU in this scenario.  2) The formula is not compared with alternative methods such as [2].

[1] Mohtashami A, Jaggi M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300. 2023 May 25.
[2] Ren H, Dai H, Dai Z, Yang M, Leskovec J, Schuurmans D, Dai B. Combiner: Full attention transformer with sparse computation cost. Advances in Neural Information Processing Systems. 2021 Dec 6;34:22470-82.
questions: questions: 1. What is the overhead of offloading?
limitations: limitations: The authors mention avenues for future work. However, a disucssion around overheads and better/worse inference speed is missing.
paper_summary: paper_summary: This paper focuses on the issue of extending context windows in LLMs by proposing a training-free block memory retrieval module. Specifically, aside from retaining initial tokens and local window tokens, other tokens are recalled using KNN at the block level. For efficient inference acceleration, LRU and chunk methods are employed to manage past key values in CPU or GPU memory. The proposed approach is tested on Mistral-32K and LLaMA-3-32K/1M models using benchmarks such as InfiniteBench and LongBench, with input tokens up to 214K. Results indicate that performance is generally maintained or improved across most tasks and models. However, there are notable performance drops in certain tasks when applying Long-context LLMs, such as the Choice task in Table 2.
strength_weakness: strength_weakness: 1. The paper does not discuss the relationship and performance comparisons with other related memory-based methods[1-3], even though many require training.
2. While InfLLM performs well on most LLMs without extended context windows, Table 2 shows performance drops in some tasks for Long-context LLMs with InfLLM.
3. The datasets used in the experiments do not clearly reflect the changes in effective context window length when using InfLLM.
4. There is a lack of necessary ablation experiments to demonstrate the effectiveness of the block-level design.
5. The paper lacks results on end-to-end latency across different context windows.

- [1] Unlimiformer: Long-Range Transformers with Unlimited Length Input, NeurIPS 2023.
- [2] Focused Transformer: Contrastive Training for Context Scaling, NeurIPS 2023.
- [3] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.
questions: questions: - **Q1**: Do you have any discussions or experimental results using other memory-based methods[1-3]?
- **Q2**: Do you have results using other well-trained Long-context LLMs, like Yi-200K[4]? Could you explain the significant performance drop in the Choice task in Table 2?
- **Q3**: Do you have any effective context windows benchmarks results, like Needle In A Haystack[5] or RULER[6]?
- **Q4**: Do you have the ablation results for the block-level design? Could you explain why performance drops with decoding only?
- **Q5**: Do you have the end-to-end latency results for different context windows?

[4] https://huggingface.co/01-ai/Yi-34B-200K

[5] https://github.com/gkamradt/LLMTest_NeedleInAHaystack

[6] https://github.com/hsiehjackson/RULER
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces InfLLM, which is a training-free memory-based method for long context extension. The key mechanism is to incorporate the sliding window attention with an efficient context memory. Each token only attends to local and relevant contexts from the memory. The paper conducts extensive experiments to demonstrate the effectiveness.
strength_weakness: strength_weakness: 1. Although the paper conducts many experiments on LongBench and InfiniteBench, the two benchmarks do not reflect long capabilities. I suggest that the paper should also include ruler[1] and long context code understanding benchmarks[2]. 

2. The author makes comparisons with the training-based long context extension method and RAG in Sections 4.4 and 4.5 respectively, concluding that they can achieve comparable results but with better efficiency. However, the baseline models chosen by the author are not strong enough. Thus, the results and conclusion are not convincing. For example, there are many high-quality long context LLMs on 128k context window, please add them as baselines to evaluate InfLLM's performance within 128k context window. 

3. The idea of using the initial token (e.g., streamingLLM) and selecting relevant tokens from past tokens for long context extension is incremental.

[1] https://arxiv.org/abs/2404.06654

[2] https://evalplus.github.io/repoqa.html
questions: questions: See the weakness section
limitations: limitations: The main limitation of this paper is that the author claims that the training-free long context extension method can reach a 1M context window, and is comparable to training-based methods. This is counter-intuitive, and based on my own experience, long context extension requires fine-tuning for real application usage. However, there is no theoretical proof or analysis to support this, and the empirical experiments are not sufficient.
paper_summary: paper_summary: This paper proposes a key/value retrieval-based method for long-context processing in Transformers. To address the distraction issue in long-context processing, the authors introduce an efficient method that retrieves blocks of relevant keys and values for attention computation. Through extensive evaluations on long-context processing benchmarks, the authors demonstrate the efficiency and effectiveness of the proposed method compared to existing state-of-the-art baselines.
strength_weakness: strength_weakness: - The paper does not introduce new technical challenges, methods, benchmarks, or phenomena. 
  - As I understand it, the proposed method adds a retrieval technique based on techniques from previous works (e.g., sink token, reassignment of positional encoding [1]). 
  - Additionally, the block-wise attention mechanism is not new (e.g., landmark attention [2]), and some methods like the kNN-Transformer use advanced retrieval techniques such as Faiss [3], which can cluster and construct blocks with similar elements. 

While I believe the paper is practically useful, the lack of novelty leads me to assign it a moderate score as a research paper.

[1] "Efficient Streaming Language Models with Attention Sinks", ICLR 2024  
[2] "Landmark Attention: Random-Access Infinite Context Length for Transformers", NeurIPS 2023  
[3] "Memorizing Transformers", ICLR 2022
questions: questions: - Are the representative tokens different across layers? Can you observe any interesting patterns or insights regarding the representative tokens?
- I am not convinced why a memory unit size of around 100 achieves the best performance in Figure 2-(c). (i.e., Wouldn't the factual/contextual information be contained in fewer tokens than that?) In the figure, the authors fix the total context size, so a smaller memory unit size retrieves more memory units. What happens when we fix the number of retrieved units and vary the size of each unit?
limitations: limitations: Yes, in Appendix.

Comments: title: Thank you for rebuttal
Thanks for your rebuttal
Looking Forward to Further Discussion
Looking Forward to Further Discussion
Looking Forward to Further Discussion
Official Comment by Reviewer 5j15
comment: Thank you for your comments and critique regarding our manuscript. We appreciate your perspectives and acknowledge your concerns about the necessity of a training-free setting. The training-free paradigm is one that has garnered significant attention within the research community. Our decision to adopt this setting was informed by its emerging relevance and several unique benefits that we have detailed in our paper and responses. We believe this setting holds considerable potential for future developments in our field. We respect and value your opinion and we plan to expand our discussion on this topic in the revised manuscript. Moreover, we are eager to further explore these issues with you in person at the upcoming conference, as we believe they present valuable and promising avenues for research.
Dear Authors,

I agree that extrapolation is definitely an important challenge. But there is no need to achieve extrapolation without training as long as you do not need to train at the target context length. There are several earlier work that require a bit of fine-tuning but then allow the model to extrapolate. So in these cases, you also can train at a context length such as 1024, and then do inference at 128k. One example is Memorizing Transformers [1]. I think not needing the fine-tuning is a nice bonus but unlike how you are trying to frame it, it is absolutely not essential to the problem of extrapolation. Again, since you keep emphasizing this point, I understand that training at the target inference length is not considered extrapolation and that is not what I am talking about here.

> Formula of Our Approach: Given the importance and challenges of training-free context extrapolation, suggesting a change in the formulas of InfLLM may not be a good choice. Our contributions are specifically within the context of training-free extrapolation, a key research direction parallel to training-based methods.

I am baffled by this response. Why is that not a good choice to suggest showcasing the importance of the chosen formula and what does it have to do with the importance of extrapolation?

> We compared InfLLM with full-attention models in the original paper and global response. Full-attention model can be regarded as the upper bound of long-text models. Our experiments on InfiniteBench have shown that our approach can achieve comparable performance with full-attention methods, even without training.

Yes but it is not clear whether the set of tasks you have chosen is suitable for correctly demonstrating long-context performance. This is an outstanding challenge to find good benchmarks for long-context. In the meantime, providing comparison with other long context methods can show how hard the task actually is and also provide the reader with a way to evaluate what amount of loss in precision is acceptable. For example, if all methods are able to get 99%, losing 0.1% accuracy is significant. Whereas if most methods only get 20% and full attention gets 90%, even a 70% accuracy is considered impressive.

> The RULER experiment was conducted using Llama3 and focused on multi-step reasoning tasks in long texts, which explains the relatively lower accuracy.

Can you elaborate on how using LLama3 explains the lower accuracy? the RULER paper reports a >80% average for LLaMA3.1 (8B) so I am not sure if the claim that the problem is the model not being able to do multi-step reasoning, is reasonable.

> Offloading experiments
Thank you. This is indeed interesting and useful.

I am maintaining my score given the above discussion.
Thank you for your feedback. As you noted, within certain limits, increasing the overall amount of relevant content retrieved does lead to better performance. However, as demonstrated in Figure 2(b) of the original paper, performance across all three tasks begins to decline when the selected units reach 128. This is due to the presence of increased noise within the context. Blindly increasing the content in the context does not always yield better performance.

Regarding the memory lookup performance of InfLLM, we wish to emphasize, as detailed in Section 4.4, that InfLLM achieves comparable results to Llama3-1M with a 128K window using only an 8K window—just 6.25% of the window size. InfLLM has advanced the processing of long texts in a training-free setting with a limited context window effectively. However, we must acknowledge that InfLLM still requires several tens of retrieved blocks to achieve optimal results, indicating significant room for improvement in precisely locating the relevant memory units. Future exploration is needed, including dynamic block segmentation and more effective block representation.

Nonetheless, these challenges do not negate the contributions of InfLLM. InfLLM provides an approach to handling long texts in a training-free manner: consistently utilizing a limited context window to select and retain crucial information from long sequences.

I appreciate your attention to these points and look forward to your reconsideration based on these clarifications. We will add the discussion in the revision.
Thank you for your comments. I would like to re-emphasize the following points:

1. **Training-Free Context Extrapolation**: You mentioned that training-free is "a slightly different setting." I cannot agree with this view. As we discussed in the global response, training-free context extrapolation is a challenging and crucial task requiring further efforts and is parallel to training-based context extension. As large models continue to grow, the ability to extrapolate the context window size without training has become a crucial area of focus, receiving significant attention [1-6]. The most relevant works to our paper are those that apply sliding window attention in a training-free manner, such as LM-Infinite (NAACL 2024 Outstanding Paper) and StreamingLLM (ICLR 2024), both of which have garnered considerable attention and follow-up. All contributions in our paper are aimed at advancing the issue of training-free context extrapolation, not just improving the efficiency of LLMs in long text inference. We agree that memory-based models are widely applied in language models, but their application without training remains a significant, largely unexplored challenge.

2. **Importance of Training-Free Extrapolation**: As stated in our introduction and global response, training-free context extrapolation is a **crucial and challenging task**. We wish to further emphasize its importance:
   - It holds potential for **real-world applications requiring unlimited streaming inputs**. As mentioned, any training-based model encounters challenges with out-of-distribution scenarios caused by unseen context lengths. Even a 128K LLM struggles in these streaming contexts.
   - Training on long texts can adversely affect model performance on shorter texts [7]. Many studies have shown that training on long texts impacts short-text performance, a problem avoided in training-free settings.
   - Continuing to pre-train on long texts is not the optimal approach. Creating diverse, high-quality long-text alignment datasets remains extremely difficult. For instance, even after extended pre-training of the Llama3-base model on long texts, we still lack the high-quality alignment data needed to achieve results comparable to Llama3-instruct.

   To summarize, training-free context extrapolation is an important method for most researchers to get a well-performing long-text model. At the same time, all the above points cannot be solved by training-based methods.

3. **Formula of Our Approach**: Given the importance and challenges of training-free context extrapolation, suggesting a change in the formulas of InfLLM may not be a good choice. Our contributions are specifically within the context of training-free extrapolation, a key research direction parallel to training-based methods.

4. **Model Performance**: We compared InfLLM with full-attention models in the original paper and global response. Full-attention model can be regarded as the upper bound of long-text models. Our experiments on InfiniteBench have shown that our approach can achieve comparable performance with full-attention methods, even without training. Thus, we argue that our comparative experiments are sufficiently robust, demonstrating that our method approaches the effectiveness of full-attention models in a training-free setup. Therefore, compared to training-based memory method, InfLLM can achieve at least comparable performance with no additional training and can be applied in a plug-and-play manner for all transformer-based models, demonstrating its practicability. 

5. **RULER Experiment**: The RULER experiment was conducted using Llama3 and focused on multi-step reasoning tasks in long texts, which explains the relatively lower accuracy.

6. **Offloading Experiment**: We have supplemented our paper with additional experiments on offloading. Here we present the relationship between the size of GPU cache and inference time costs of Passkey Retrieval. From the results, we can observe that the offloading mechanism will not incur much additional time costs with limited cache size.
   | Size of GPU Cache | All   | 32    | 16    | 0     |
   | ----------------- | ----- | ----- | ----- | ----- |
   | Time (s)          | 28.80 | 28.89 | 29.33 | 33.50 |

I appreciate your attention to these points and look forward to your reconsideration based on these clarifications. We will add the discussion in the revision.

[1] Efficient streaming language models with attention sinks. ICLR 2024.
[2] LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. NAACL 2024 (Outstanding Paper).
[3] LLM maybe LongLM: Self-extend llm context window without tuning. ICML 2024.
[4] Training-Free Long-Context Scaling of Large Language Models. ICML 2024.
[5] Scaling laws of rope-based extrapolation. ICLR 2024.
[6] A human-inspired reading agent with gist memory of very long contexts. ICML 2024.
[7] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. ICML 2024.
In the rebuttal, the authors argue that "as the unit size increases, the overall amount of relevant content retrieved also increases, leading to better performance" (up to 512 tokens, which span approximately one page). However, it remains unclear why such an long context length is necessary to achieve the desired performance in their experiement settings. In L42 of the manuscript, the authors claim that "processing each token typically requires only a small portion of its context, and the remaining irrelevant context acts as noise, leading to attention distraction issues." I’m not convinced that their proposed method actually resolves this problem—it might simply be discontinous/OOD positional encoding issues. Given the weaknesses I've mentioned, I maintain the current score.
Dear Authors,

Thank you for your replies. 

Regarding comparison with existing methods: I understand that the cost of running some of these methods can be quite high. I also understand that proposing a method that is showing good performance can be generally useful. On the other hand, there is a growing number of proposed methods (each possibly targeting a slightly different setting, e.g. training-free) all of which report great accuracy on some tasks.  But without any comparison or ablation study it is impossible to identify the advantages of the proposed method. 

I made specific suggestions in my review such as using alternative formulas instead of the proposed formula, particularly those used in previous work such as [2] in my review (though [2] does not perform training-free lookup, the formula can still be used). The authors did not provide any reply on this suggestion. The idea is to understand whether that specific formula matters. 

Also thank you for pointing out that you are matching the full attention in Table 2 but this does not answer whether other methods would do so as well or not. In other words, it is not clear whether the task is not hard enough to distinguish the differences or the method is good.

> Additionally, InfLLM is the first work to design a memory module specifically suited for training-free context extrapolation, implementing a comprehensive and efficient inference solution. 

I am not sure if this claim is justified. Similar not fully justified claims are often made throughout the paper as I mentioned in my original review. The idea of maintaining a cache and offloading to CPU is not novel and has been implemented before, for example in [3]. Various eviction policies have also been looked at for example in [4]. The combination with retrieving based on a representative has also been looked at before. That is what the memory module is doing.  Can you please specify what is the special novelty of the memory module that makes this the first work to design it? The low-level implementation is useful and I do not know any other work that has done it. But then this module needs to be fully benchmarked to show the impact on inference speed.

As I mentioned in my original review, in my view the paper's novel contributions seem to be "the specific formula used to obtain the representative tokens, and the low-level offloading implementation together with cache" and it should be clearly framed as such. That being said, I do not think these are light contributions and once they are fully justified to be crucial to good performance, they yield a very impactful result.

Also thank you for running the InfLLM on RULER but the numbers that reported are much lower than those reported in the original paper's Table 1. For example Mistral 7B obtains the average 68.4. What is the base model in this case?

Regarding the offloading: Thank you for the explanation and reporting the effect of the cache. I think this will be a useful addition  to include in the paper. However, what I was asking about is the overhead of offloading itself. To be more precise, the question is what is the time difference between keeping everything on GPU and offloading to CPU (either with or without cache). I understand that as the length grows, keeping everything on GPU becomes impossible but I still want to understand what is the overhead of moving things to CPU. 

[3] Aminabadi RY, Rajbhandari S, Awan AA, Li C, Li D, Zheng E, Ruwase O, Smith S, Zhang M, Rasley J, He Y. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis 2022 Nov 13 (pp. 1-15). IEEE.

[4] Zhang Z, Sheng Y, Zhou T, Chen T, Zheng L, Cai R, Song Z, Tian Y, Ré C, Barrett C, Wang Z. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems. 2024 Feb 13;36.
Thanks for your detailed response! Although InfLLM performs better than streamingllm on the ruler, its performance on the ruler is still far behind other training-based long context approaches. I believe this might indicate that InfLLM is limited in practical long-context applications, so I will keep my original rating score.
Dear Reviewer,

Thank you for your ongoing engagement with our paper. As the discussion period unfolds, we welcome any new concerns regarding our responses. We were pleased to learn from Reviewer 5j15 that our explanations have adequately addressed key issues, including comparisons with well-trained long LLMs, broader benchmark evaluations, and the innovation presented in InfLLM. We eagerly anticipate your further feedback on these topics. If our replies have resolved your initial concerns, we kindly request an improvement of your initial scoring. We appreciate your commitment and time invested in reviewing our work.
Dear Reviewer,

Thank you once more for your insightful feedback on our paper. The discussion period is now underway, and we welcome any additional questions regarding our responses. Reviewer 5j15 has acknowledged that our explanations addressed concerns about comparisons with memory-based methods and the novelty of InfLLM. We are keen to hear your thoughts on these issues as well. If you find that our responses have sufficiently addressed your concerns, we would appreciate a consideration for an improved score. Thank you again for your dedication and time.
Dear Reviewer,

Thank you once again for your valuable comments on our paper. The discussion period has now started, and we welcome any further questions you may have regarding my responses. Reviewer 5j15 has indicated that our responses have resolved his/her concerns regarding the comparison with memory-based methods. We are also eager to hear your views on this matter. Should our responses have addressed your concerns, we would be grateful for an improved score. Thank you again for your time and efforts.
Thank you for your thorough and insightful feedback on our work. We greatly appreciate your kind words and the time you took to review our paper. Your positive comments and constructive suggestions have been invaluable in refining our research. We're glad that our efforts to build memory modules for enhanced long-context capabilities resonated with you.

Thank you once again for your support and encouragement.
Thank you to the authors for the detailed rebuttal! I truly appreciate your efforts in putting everything together within such a short period.

My concerns have been addressed, and I have read the questions and answers from other reviewers as well. I have no further concerns. Overall, I believe InfLLM is a commendable work. It leverages the characteristics of natural language to design a block-wise, training-free retrieval method to extend long-context capabilities, resulting in improved latency and performance. The motivation is clear and the experiments are thorough.

I have raised my score to 7.