Core Context Aware Transformers for Long Context Language Modeling
Authors: Yaofo Chen
Zeng You
Shuhai Zhang
Haokun Li
Yirui Li
Yaowei Wang
Mingkui Tan
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models; Efficient Attention
Primary area: deep_learning->large_language_models
TL;DR: We propose Core Context Aware Attention that greatly reduces the context redundancy with strong performance on long context modeling.

Abstract: Transformer-based Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to self-attention mechanism, which requires a token to consider all preceding tokens as its context to compute attention. However, when the context length L becomes very large (e.g., 128K), the amount of potentially redundant information in the context tends to increase. The redundant context not only hampers the modeling representation performance but also incurs unnecessary computational and storage overhead. In this paper, we propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling, comprising two complementary modules: 1) Globality-aware pooling module groups input tokens and dynamically compresses each group into one core token based on their significance. In this way, our method automatically focuses and strengthens core context while diminishing redundancy during the learning process, leading to effective long-term dependency modeling. 2) Locality-preserving module incorporates neighboring tokens to preserve local context for detailed representation. Notably, our CCA-Attention is able to replace the self-attention module in existing LLMs with minimal fine-tuning cost. Extensive experimental results show the superiority of our method in both long-context modeling and computational efficiency over state-of-the-art methods.

Core Context Aware Transformers for Long Context Language Modeling
Yaofo Chen
Zeng You
12
Shuhai Zhang
Haokun Li
Yirui Li
Yaowei Wang Mingkui Tan 135
Abstract
1. Introduction
Large language models (LLMs) (Brown et al., 2020; Ope- nAI, 2023; Touvron al., 2023a; Liu et al., 2024a) have demonstrated exceptional proficiency across various appli- cations by effecttivly modeling extended contexts, particu- larly in tasks involving natural language understanding and generation (Ouyang et al., 2022; Chang et al., 2024). The remarkable success of Transformer-basee LLMs is predom- inantly credited to the self-attention mechanism (Vaswani et al., 2017), which requires each token to incorporate all preceding tokens as its context for attention calculation. In this mechanism, the context of token refers the sequence of tokens that precede it. By leveraging self-attention, LLMs are able to capture long-range dependencies and generate coherent and contextually relevant outputs.
Transformer-bssed Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to the self- attention mechanism, which requires a token to consider all preceding tokens as its context to com- pute attention. However, when the context length L becomes very large (e.g., 128K), the amount of potentially redundant information in the con- text tends to increase. The redundant context not only hampers the modeling representation perfor- mance but also incurs unnecessary computational and storage overhead. In this paper, we propose a plug-and-play Core Context Aware (CCA) Atten- tion for efficient long-context modeling, compris- ing two complementary modules: 1) Globality- aware pooling module groups input tokens and dynamically compresses each group into one core token based on their significance. In this way, our method automatically focuses and strength- ens core context while diminishing redundancy during the learning process, leading to effective long-term dependency modeling. Locality- preserving module incorporates neighboring to- kens to preserve local context for detailed repre- sentation. Notably, our CCA-Attention is able to replace the self-attention module in existing LLMs with minimal fine-tuning cost. Extensive experimental results show the superiority of our method in both long-context modeling and compu- tational efficiency over state-of-thee of-the-art methods.
The ability to process longer contexts has been a key fac- tor in improving the performance of LLMs across wide range of tasks, particularly beneficial for tasks requiring document-leved understanding, such as summarization of extended texts, and multi-turn dialogue (Kitaev et al., 2019; Wei et al., 2022; Jiang et al., 2024). More importantly, recent advancements in LLMs, such as OpenAl-ol (OpenAI, 2023) and DeepSeck-RI (Guo et al., 2025), have demonstrated that extended contexts significantly enhance reasoning ca- pabilities, enabling models to solve intricate problems that require multi-step inference and contextual understanding. This trend underscores the importance of extending the con- text length in LLMs to enhance modeling capabilities.
However, as the context length L. scales to extremely large magnitudes (e.g., 128K), it becomes impractical for a to- ken to maintain significant semantic connections with all tokens within such an extensive context. From this per- spective, is natural to consider that not all parts of the context contribute equally to token's representation. In- stead, the context can be viewed as comprising two primary aspects: core context, which captures essential semantic connections, and redundant context, which contains less critical or repetitive information. The redundant context may hamper LLMs from capturing dependencies among erucial tokens, degrading representation performance. In self-attention, this redundancy manifests as a highly sparse distribution of attention scores, with substantial proportion disproportionately assigned to a limited number of tokens.
'ggual contribution 'South China University Technol- ogy PPng Cheng Laboratory 'Pazhou Laboratory "Harbin Institute of Technology "Key Laboratory of Big Data and Intelligent Robot, Ministry of Education. Correspondence to: Mingkui Tan <mingkuitan& scut.edu.cn>>, Yaowei Wang <wangyw @ pww..c.cc.c.......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Core Context Aware Transformers for Long Context Language Modeling
Such sparsity in attention score distribution has been ob- served across different attention heads in most layers of LLMs, as shown in recent studies (Beltagy et al., 2020; Zaheer et al., 2020; Xiao et al., 2024b). This sparsity in attention scores introduces unnecessary computational and storage overhead, especially for extremely long contexts.
We propose a plug-and-play Core Context Aware At- tention for efficient long-context modeling. Our CCA- Attention reduces computational complexity to linear complexity by taking a set of core tokens as efficient proxies for attention. Unlike traditional efficient at- tention methods that require extensive retraining, our CCA-Attention can be easily integrated into pretrained LLMs with minimal fine-tuning effort.
To address the above issues, numerous studies have been advanced to eliminate the redundancy and enhance atten- tion computational efficiency. StreamingLLLM (Xiao al., 2024b) and LM-Infinite (Han et al., 2023) simply main- tain the attention over only the initial and last several to- kens, ignoring the attention connection among remaining tokens. Besides, MInference (Jiang et al., 2024) introduces an efficient mixed attention mechanism comprising A-shape, vertical-slash, and block-sparse attentions, with the mixed attention patterns determined offline based on some samples. These methodologies typically involve computing only portion of the attention to approximate full attention, thus compromising the connections among different tokens. In question-an wanhiiinn tasks, the crucial information can be lo- cated across any position in the input tokens. Consequently, it is crucial for the model to have the capability to leverage information from any position within the input text (Liu et al., 2024b). In this sense, these methods with fixed spar- sity patterns may lead to incomplete comprehension of the long context. Therefore, how to ensure the information ex- change among tokens in the attention while reducing the context redundancy is still an open question.
We develop a dynamic globality pooling mod- ule that adaptively derives core tokens based on their importance. By compressing input tokens into core to- kens, our method captures essential information more effectively than static or random token selection ap- proaches. Our strategy focuses on the most relevant global context, leading to more accurate and effective long-term dependency modeling.
We achieve significant improvements compared with other baseline methods in both long-context modeling performance and computational efficiency. Our ex- perimental results show that CCA-Attention not only outperforms existing efficient attention mechanisms in long-context modeling but also achieves 7.9x speedup compared to full self-attention when process- ing 128K token contexts, demonstrating substantial efficiency gains with compatible accuracy.
2. Related Work
In this paper, we propose an efficient Core Context Aware (CCA) Attention mechanism, which is designed to effi- ciently capture both global and local dependencies within long contexts. Specifically, our CCA- two complementary components: 1) Globality-awre pool- ing module first partitions the input tokens into groups and derives core tokens by compressing the input tokens in each group based on their significance. We perform attention on these core tokens instead of original input tokens to ef- ficiently extract long-term contextual information. These number-reduced core tokens are more compact represen- tations than the original ones, which enables our attention method to automatically focus on the core context. In this way, our method is able to eliminate the context redundancy and reduce unnecessary computational overhead. However, the globality-aware pooling module is only able cap- ture long-range and coarse-gie gaatse-rainedddd information. 2) To ad- dress this issue, we propose prcaiit--reseervigg module that captures the local and fine-grained context by focus- ing on neighboring tokens, serving as a complement for the globality-awe awaar pooling module. By fusing the insights from both these two modules, our method not only excels in long context modeling but also achieves this with a signifi- cant reduction in computational costs and storage demands. Our contributions are as follows:
Efficient Attention. ule in Transformer-based Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023a). captures the global relationship between each token through- out the input sequence. However, the computational com- plexity of self-attention increases quadratically with the sequence length, thereby limiting the application of LLMs to long documents. Various works have sought to mitigate this complexity through approaches such as sparse attention (Beltagy et al., 2020; Zaheer et al., 2020; Ding et al., 2023) and linear attention approximations (Choromanski et al., 2020; Katharopoulos et al., 2020; Sun et al., 2023). Specifi- cally, Longformer (Beltagy et al., 2020) and BigBird (Za- heer et al., 2020) employ sparse attention mechanisms to handle long sequences by utilizing strided attention patterns, where attention is only paid at fixed intervals. Linear Trans- former (Katharopoulos et al., 2020) and RetNet (Sun et al., 2023) reformulate self-attention as a linear dot-producc of kernel feature maps and leverages the associativity property of matrix products to achieve linear complexity.
Self-attention is a fundamental mod-
consists of
Recently, LongLoRA (Chen et al., 2024) designs a shifted sparse attention mechanism that computes attention among grouped input tokens. To facilitate communication between groups, this approach shifts the group partition by half the
2

---

Core Context Aware Transformers for Long Context Language Modeling
LLM
computational
model
capable
of
language
generation
other
natural
language
processing
tasks
(Rxx)(Redundant Context)
Corre Context)
(Crrr Context)
Reduundnt Context)
Figure 1. Illustration of core contexts and redundant contexts. We show attention scores of the last token relative to the other tokens in LLaMA2-7B (darker shadows indicate higher attention scores). The last token exhibits high attention scores towards core contexts. The remains are considered as redundant contexts, introducing unnecessary computational overhead for attention.
group size. StreamingLLM (Xiao al., 2024b) and LM- Infinite (Han et al., 2023) prioritize attention on the initial and final tokens, effectively disregarding the intermediate to- kens. InfLLMM (Xiao et al., 2024a) employs a sliding window attention mechanism and a block-level context memory to selectively attend to relevant context information, avoiding noise and reducing computational costs. MInference (Jiang et al., 2024) accelerates long-context LLM inference by applying three distinct sparse attention patterns with opti- mized GPU kernels. However, these methods can not ensure that each token has access to all preceding tokens, leading to inferior performance in tasks requiring comprehensive long-context understanding. Instead, we propose a globality- aware pooling module that each token can communicate with previous tokens via number-reduced core tokens.
redundant information. Unlike sequence-length-oreee -rriened compression techniques, our crre-contettawwal mechanism optimizes redundancy directly within olf-attention compu- tation, enabling more effective long-context modeling.
3. Core Context Aware Attention
3.1. Motivation and Method Overview
Most existing attention-based large language models (LLMs), such as GPT (Brown et al., 2020; OpenAI, 2023) and LLaMA (Touvron et al., 2023a), employ the next-token prediction (Vaswani et al., 2017) paradigm to generate text. Given a sequence of tokens each token x; E 1 the model predicts the next token x; by conditioning on all preceding tokens as its context C(x,)=(xx:: Specifically, the model generates the next token with the highest probability as:
X2; Xx), where
Long-context Large Language Models (LLMs). L.LSs are often pretrained with a relatively small and predefined context length due to computational cost constraints, such as 4K for LLaMA2 (Touvron et al., 2023b). This limita- tion restricts the applicability of LLMs to tasks with long documents. Recently, several attempts have been made to extend the context length of LLMs through continuous train- ing, Position Interpolation (Chen et al., 2023) addresses this by linearly down-scaling the input position indices to fit within the original context window size, thereby extend- ing the context length of RoPE-basee LLMs. Furthermore, YaRN (Peng et al., 2024) enhances performance by combin- ing interpolation techniques with dynamic scaling. Beyond modifications position embeddings, other efforts focus designing more efficient attention mechanisms (Chen et al., 2024; Dao et al., 2022; Dao, 2024) for context window ex- tension. Our method is orthogonal to position embedding methods. During inference, our approach accelerates the forward propagation process, which cannot be achieved through position embedding modifications alone. Some context compression works attempt achieve long con- text modeling by either compressing features via auxiliary networks (Rae et al., 2020) or compressing context with extra specific tokens (Mu et al., 2023; Qin & Van Durme, 2023; Mohtashami & Jaggi, 2023; Zhang et al., 2024; Qin et al., 2024). In contrast, our method dynamically identifies and enhances task-relevant core-context while suppressing
x; =arg max Po (x xx-1)..
(I)
However, as the context length L grows, the context in- evitably exhibits redundant information. This redundancy stems from the inherent nature of natural language: not all contextual information is equally important for the rep- resentation of the target token. These redundant context (e.g., C2l (x,) w.t. x, in Figure 1) have weak semantic rele- vance to the token x,, while introducing significant compu- tational overhead. In contrast, the core context (e.g., Cx(()) w.E.l. x, in Figure 1) refers to the contextual information that is highly relevant the token x,. This information is crucial for the token's 's representation. Therefore, in long-context modeling, the model should prioritize the core context over redundant parts. To this end, most existing methods (Belt- agy et al., 2020; Ding et al., 2023; Chen et al., 2024) employ sparse attention with predefined and fixed patterns. Unfortu- nately, they often overlook the importance of maintaining comprehensive information exchange among tokens, which may hinder the performance of long-context modeling tasks.
In this paper, we seek to reduce the context redundancy asso- ciated with full self-attention. To achieve this, we propose a Core Context Aware Attention (CCA-Attention)) which em- plays globality pooling and locality preeerrii mod-
3

---

Core Context Aware Transformers for Long Context Language Modeling
Input Token Sequence X 11
Globality-aware pooling Module
Group-wise Weighted Pooling
17t
Grouping with size
Sliding with window size
9
token grouping
window-sliding
CE Rond
9
K IRaxd
9
8
X R
A R
Hlobaliit--wwar poolingg Module
2
Lecatity-reserving Module
m [L/g]
(XW Daxt
Complexity: (Zm)
KG
VG
K'
v
b.ocality-reseeviing Module
d'
Sliding with window size
Q B dxl.
Att R
Differentiable Fusion of Global and Modules
Att softmax [Q(Kk; K-] ( vv
K' g
R
g2xdd
V R
Complexity: (1.ss)
Figure 2. Illustration AECA-Atention, which includes two components: 1) Globality pooling module encapsulates the input tokens X into tokens C according to the importance (Eqn. (2)). The tokens serve representative proxies of X for attention, thereby reducing computational costs. 2) Locality-presservnn module incorporates the local context from neighboring tokens, acting as supplement for the globality pooling modulc. We produce the final output Att by fusing these two modules based on Eqn. (5).
ules to capture both global and local dependencies within long context. As shown in Figure the globality-aware pooling module operates by generating representative core tokens from segmented groups of the input sequence. It then computes attention using these reduced-number core tokens, thereby reducing the context redundancy and computational cost (see Section 3.2). However, the globality-awe pooling module mainly focuses on long-range and coarse- coarse-graing graiined information and overlooks local context. To address this limitation, the locality-preserving module is responsible for capturing the local information of the neighborhood to- kens to ensure comprehensive coverage (see Section 3.3). Furthermore, we devise a differentiable fusion strategy to combine the insights from global and local modules (see Section 3.4). This is crucial as it retains the comprehen- sive understanding ability of the full self-attention within CCA-Attention. The pseudo-code for our proposed is presented in Algorithm
segment the input sequence X, each group containing g tokens, total m= vt=[L/g] groups. For simplicity, we de- note the i-th group by X ERoxd where (--(i- + 1, (i-1)g+2,.... ;ig) with Xig denoting the last token in the i-th group. To identify prominent tokens in the i-th group, we devise a group-wise weighted pooling strategy that em- ploys the last token Xig to evaluate the importance. This is inspired by attention map visualizations (Section C.4), which show that important tokens consistently receive high attention scores from subsequent tokens, indicating their significant influence regardless of position within the group. Formally, we derive one core token c, from each group by
Qig K
c, =;=softmxx
X E Elixd 1,
where Qig is the query vector for the last token in the th group of =z,=xē wº and K =X WK, WQ and W are learnable parameters. In this way, the core token c; encapsulates crucial information of the corresponding group. With m groups in the input sequence X, we derive m core tokens in total, i.e., C=(c1; ;
3.2. Globality-aware Pooling Module
The context redundancy aforementioned indicates that com- putational resources can be dynamically allocated to core contexts while reducing emphasis the remaining ones. This could approximate the full self-attention with both re- duced redundancy and computational overhead. Motivated by this, we propose a globality-awe -waare pooling module that dynamically identifies prominent tokens and encapsulates them into smaller set of core tokens for attention.
To reduce the redundancy, we use the sequence of core to- kens C=(c1; C2;.... : Cm] instead of the original tokens X for attention computation. This substitution reduces the di- mensionality from X Lxd to CeRmxd, thereby reducing the computational and storage complexity. For each query Q, tokens that are distant from it typically exhibit lower relevance and are more likely to be redundant. Formally, we adopt core tokens calculate the key and value matrices
Given input sequence of tokens X= Xxx;;XX;- ...; Xr.,, we

---

Core Context Aware Transformers for Long Context Language Modeling
Algorithm 1 The pipeline of Core Context Aware Attention.
window size 8+ s+((i- - s) mod g). This strategy ensures that each key token participates in the attention computation with the query Qj Consequently, the key and value matrices
Require: Input tokens X=(x1; X2; : xx,], parametens W
W WV the group size y and local window size s.
1: Calculate the query Q = XW?, #groups m = [T//]]
for specific query Q; in the wecality-preservngg module are defined as follows:
2: for i in (1, 2....,,......... m) do
3: X2. (4 1)0 Xigl
K K. VI, - Vil. K 14
4: c==softmax
Va
I
X9 where K2 =2 W
K-= XW VL XWV
(4)
5: end for
6: Let C = [0:;0::.....
where k= max ((i-s())) mod g)). Note that the locality module shares the projection parameters W W and WV with the globality awwre pooling mod- ule, thereby incurring no additional projection parameters.
7: -WW -C-CW Globality-aware Pooling Module
8: K" XW V XW Loccditypreserving Module
9: for in (1,
D} do
10:
KG K-,-KK - 1:j1 Vc = Vigg j= max(0, ((1-)//g1)
11:
KW. K
V Rid k= (0-(1))--(((i))) mod g))
3.4. Differentiable Fusion of Global and Local Modules
12:
Atl: = softmax Q
K Kt, (ttll)///)(V //VV)(VV Vigl
13:
end for
Both globality-awar pooling and ules involve only a portion of tokens in the attention compu- tation, leading to a limited comprehensive understanding of the context. To address this limitation, we seek combine the involved tokens of these two attentions to integrate the insights they provide. Specifically, we concatenate the key and value matrices from both attentions, i.e., [K4; K4, and [VG; V4,, to leverage the combined information. For- mally, the proposed CCA-Attention is computed follows:
mod-
14: Return: Representations of tokens Att= Att Attz]
K yu for these tokens as follows
KG - K VG V ;Vi, CW vV_CwV
(3)
where W and W is learnable parameters. In contrast, tokens in close proximity to the query Q are likely to be more relevant. We retain s nearest tokens for a fine- grained attention computation (discussed in detail in Sec- tion 3.3). Thus, the index j in Eqn. (3) can be calculated as j= max(0, ((i-)) When the context is short (i.e., i<(g+ + *)), the key and value K VG would be excluded from attention calculation since the redundancy in the con- text is negligible. During inference, as tokens are generated sequentially, we derive a new core token via Eqn. (2) once the number of generated tokens reaches g. Different from the full self-attention, we cache KG and VG for inference.
Q KG T U4
Att, = softmax
IVO T V.
(5)
Vd
We represent the final output of CCA-Attention as Att [Att;; Att Attr]. In practice, we imple- ment our attention mechanism with Triton (Tillet et al., 2019) to accelerate both training and inference processes. This implementation enables parallel computation of each Att, with high computational efficiency. After integrating the global and local attention, we can formalize Att in Eqn. (5) element-wise into the structure of full attention, de- tailed in Proposition I in the supplementary material A. The Proposition 1 shows that each token accesses all preceding tokens, ensuring full information exchange among tokens and thus enhancing capturing long-range dependencies.
3.3. Locality-preserving Module
As mentioned above, the globality-awart pooling module ef- fectively captures long-range dependencies by compressing input tokens into core tokens. It focuses coarse-grained global information, potentially overlooking fine-grained lo- cal context. However, recent studies (Manakul & Gales, 2021; Yang et al., 2021) demonstrate that local context plays a critical role in many language modeling tasks. To address this, we introduce
More importantly, our CCA-Attention demonstrates dy- namic flexibility through adjustable group size g and local window size s during inference. This architectural flexibility allows the generation of multiple model variants tailored to varying user traffic, offering a substantial advantage over the full self-attention mechanisms in real-world deployment scenarios (see results and analysis in Section 4.3).
peeeitt-ppeervingg module that comple- ments the globality-aware pooling module by focusing on neighboring tokens to capture detailed local dependencies.
3.5. Training Strategies of CCA Transformers
To be specific, the Sccaliy-preserving module ensures that each query Q attends to the preceding at least tokens to capture local dependencies. During the generation process, it is challenging to maintain the number of tokens as a multiple of the group size g. To address this, we set the local
Our proposed CCA-Attention is fully compatible with ex- isting attention-baae LLMs, such as the LLaMA series models (Touvron et al., 2023b; AI, 2024), serving as a plug-and-play module that can replace the full self-attention.
5

---

Core Context Aware Transformers for Long Context Language Modeling
CCA-Attention maintains alignment with full self-attention in terms of input, output, and parameter dimensions. This ensures that only minimal training cost is able to pre- serve long-come mong-conext modeling capabilities while reducing computational costs. In contrast, existing linear attention approaches (Katharopoulos et al., 2020; Sun et al., 2023) introduce kernel functions for attention and necessitate train- ing from scratch, making them less practical for real-world applications due to their inability to leverage the extensive knowledge embedded in pretrained LLMs.
group size g as a constant, ensuring that our CCA-Attention is able to leverage key-value caching during autoregressive token generation. Once one group has certain y tokens, the corresponding core token is also determined and cached. Thus, our CCA-Attention achieves a computational com- plexity of O + Ls). The complexity analysis follows a similar pattern to the tasks with fixed-length sequences.
Acceleration through Reduced Key-Value (KV) Cache In attention-based LLMs, the KV cache leverages the autore- gressive nature to store and reuse key-value pairs, thereby significantly boosting the efficiency. The size of the KV cache scales linearly with the length of the input sequence, consuming a major part of the memory footprint during inference. The expanded KV cache would consume con- siderable memory and significant memory I0 resources. Compared with full attention's 's complexity of O(L), our CCA-Attention has storage complexity of O(L/g + s). For the globality-awar pooling module, we only retain the key and value matrices for core tokens, rather than for all original tokens. This reduces the memory requirement to OLL/g)- Besides, the maintains the key and value matrices for the preceding s tokens. The storage complexity for this component is O(s).
We replace the self self-attention module in existing attention- based LLMs with CCA-Attention, enabling compatibility with three different training strategies: 1) Training from Scratch: This strategy involves training CCA-Attentin from scratch on large-scale corpora. While this approach may yield superior performance by fully adapting the model to the proposed attention mechanism, it is computationally intensive and requires significant resources. 2) Full Finetun- ing: A more efficient alternative is to finetune all parameters of the model based on the parameters of existing pretrained LLMs. This strategy leverages the pre-trained knowledge while allowing the model to adapt fully to our attention mechanism. 3) Partial Finetuning: For scenarios where efficiency is critical, we can finetune only the learnable pa- rameters of strategy requires only a modest finetuning effort on small number of corpora, making it computationally efficient for maintaining long-context modeling capabilities. We pro- vide empirical evidence to guide the selection of the most appropriate finetuning strategy in Table 10.
module only
i.e., W W and WV This
4. Experiments
4.1. Experimental Setup
We apply our CCA-Attention and compared efficient atten- tion methods to existing pretrained LLMs. We report the performance in long context modeling and computational efficiency. We put more implementation details and ablation studies of our method in the supplementary materials.
3.6. Computational and Storage Complexity Analysis
Compared with the full self-attention, our CCA-Attention offers significant benefits in terms of computational com- plexity and key-value cache storage, as analyzed below.
Dataset & Evaluation Metrics. We quantitatively evaluate our models and compare them with other considered models on the following benchmark and metric: 1) LongBench (Bai et al., 2023) is a pioneering benchmark for the bilingual, multi-task, task, and comprehensive assessment of large language models' long context understanding capabilities. It covers multiple languages like Chinese and English, consisting of 6 major categories and 21 tasks involving various ap- plication areas. 2) Exact Match Score (EM Score) (Liu et al., 2024b) is a metric for measuring the model's ability to find the key information within a long context in a multi- document question-answen
Acceleration via Reduced Computational Complexity. Our CCA-Attention exhibits varying computational com- plexities depending on the type of task. For tasks with fixed-length sequences (such as multi-choice question an- swering). our CCA-Attention exhibits linear computa- tional complexity of ( + I.s), marking significant enhancement over the full self-attention with a complexity of O(L?). Here, we define the number of group m as a constant. For the globality- aware pooling module, the query and key matrices encompass L and m tokens, respectively, resulting in a computational complexity of O(Lm). Regard- module, each token only attends preceding g tokens at most. With I. tokens in total, the upper bound of complexity amounts to O[L(s + g))-
task, In this task, each test sample comprises a certain number of documents to reach the specified context length, followed by question about the key information inserted in context.
ing the
Implementation Details'. We apply our proposed CCA- Attention to LLaMA2-7B-32K, LLaMA2-7B-800 (Fu et al.,
For tasks with variable-length sequences (such as open- ended question answering), models generate subsequent tokens in an autoregressive manner. In this case, we set the
'The source code for this project is publicly available at http://ghhbbcmmmcceyyyacccCaCaaa
6

---

Core Context Aware Transformers for Long Context Language Modeling
Table 1. Comparisons of different models on LongRench-F. (Bai et al., 2023). The length of 95% of the test. samples in LongRench-FF is less than 31K. "FTL" denotes the latency to generate the first token in the pre- pre-filling stage. "Mem." denotes the memory footprint.
"S. QA" means single document QA, while "M. QA" denotes docti-d QA. We report the latency and memory footprint of
LLMMA2-7B-32K and LLaMA2-7B-80K within contexts of 32K and 64K on A800 GPUs, respectively.
Methods
QA
M. QA
Sum.
FS. Learning
Synthetie
Code
Avg.
FTL
Mem. (GB)
TLaMA2-7B-528 (Vanilla Self Self Attention)
2.75
1.85
12.43
66.28
0.34
48.99
22.11
9.15
35.58
Streaming (Xiao et al., 2024b)
4.75
2.94
2.97
48.20
0.66
30.16
14.95
5.75 (1.6x)
22.94 (35%.J)
LM-Infinite (Han et al., 2023)
2.04
2.33
1.98
57.45
0.3
48.46
18.76
(1.9%)
26.35 (26%)
MInference (Jiang. et al., 2024)
3.68
3.05
10.97
66.26
0.61
42.30
21.14
4.20 (2.2x)
33.52 (6%%L)
CCA-LLM (Ours)
3.63
3.98
7.79
61.79
2.64
51.36
21.86
2.59 (3.5x)
19.12 (46%..)
ML (Vonilla Self-Artentinn
3.22
2.71
3.90
64.98
0.56
59.16
22.42
32.43
60.03
StreamingLLMM (Xiao et al., 2024b)
2.07
2.32
0.37
45.03
2.67
37.17
14.94
9.04 (3.6x)
37.45 (37%..)
LM-Infinite (Han et al., 2023)
2.54
1.53
2.22
61.29
1.08
58.54
21.20
8.27 (3.9x)
41.54 (31 96.1)
MInference (Jiang et al., 2024)
2.44
3.49
4.41
64.26
0.28
57.60
22.08
8.14 (4.0%)
54.09 (10%.)
CCA-LLM (Ours)
5.62
4.34
8.99
59.60
0.48
54.40
22.24
6.42 (5.7x)
33.86 (44%.)
Table Comparisons of different methods using latest models on LongBench-E (Bai et al., 2023). We report the lateney and memory footprint of 1-AAA..--B-IIssiie 128K (short for ---.MM...----188-- in the table) and Qwen2.5-7B B-8K within contexts of 32K
on A800 GPUs. Methods
S. QA
M. QA
Sum,
FS. Learning
Synthetic
Code
Avg.
FTL
Mem, (GB)
[laMA3. 128K (Vanilla Self Attention
16.71
10.75
20.32
68.75
48.93
62.10
37.93
9.55
40.38
Minference (Jiang et al., 2024)
16.33
10.71
20.44
68.41
48.06
62.50
37.74
4.93 (1.9x)
35.95 (11%%-I
CCA-LIMM (Ours)
17.90
16.41
19.63
67.20
43.76
61.98
37.81
3.08 (3.1x)
20.63 (49%..)
Qwen2.5-7B-1288 (Vanilla Self-Attentio))
16.67
18.18
18,70
66.81
45.34
64.56
38,38
10.58
35.11
MInference (Jiang et al., 2024)
16.20
17.21
18.59
67.10
38.28
62,95
36.72
4.86 (2.2x)
32.40 (8%()
CCA-LLM (Ours)
16.91
17.07
18.60
66,89
45.50
63.52
38.08
2.74 (3,9x)
19,31 (45%.)
2024), LLaMA3.1-8B-1222 (AI, 2024) and Qwen2.5-7B 128K (Yang et al., 2024) models. We replace the full self-attention in the above LLMs with our proposed CCA- Attention. In the continuous finetuning, we adopt the SlimPajama (Cerebras, 2024) dataset, an open-source repli- cation of the LLaMA pretraining data mixture. The number of groups in globality-aware Attention is shared across dif- ferent model sizes. We finetune the full model on A800 GPUs using a micro-batch size of and a gradient accumu- lation of 8, with total of 1000 training steps. This training configuration is applicable to all model sizes and context lengths. To scale the models to long contexts, we modified the "base frequency" in RoPE from 10,000 to 500,000, fol- lowing (Cerebras, 2024; Xiong et al., 2024). See Section B.2 for more implementation details.
figuring the local branch size to 1024 and the global branch size to 16. In the case of MInference (Jiang et al., 2024), we also employ the official code implementations, configured with the official settings.
4.2. Comparisons on Long Context Modeling
Comparisons on Longbench-E. We conduct experiments Longbench-E (Bai et al., 2023) using our CCA-Attention and baseline methods, including StreamingLLM (Xiao et al., 2024b), LM-Infintt (Han et al., 2023), and MIn- ference (Jiang et al., 2024). As shown in Table 1, our CCA-LLM attains the highest average score on Longbench- E, outperforming other efficient attention methods. For LLaMA-7B-322 the average score of our CCA-LLM is higher than that of IM- (21.86 vs. 18. 18.76) and MInfer- ence 121. vs. 21.14). For the LLaMA-7B-80K model, our method consistently shows superior performance compared to alternative approaches. For instance, our CCA-LLM yields a higher EM score than LM-Infinite (22.24 vs. 21.20) and MInference (22.24 vs. 22.08). This performance advan- tage primarily stems from our global-aware pooling module, which effectively reduces context redundancy in long input sequences. Consequently, our CCA-LLM model demon- strates enhanced capability in identifying and focusing on core contextual elements, thereby facilitating more accu- rate extraction of crucial information required for question- answering tasks within the Longbench-E benchmark. No- tably, our CCA-LLMM achieves the lowest inference latency
Compared Methods. We conduct comprehensive com- parisons between our proposed CCA-Attention and sev- eral state-of-the-arr methods, including LLaMA-2 with vanilla attention, StreamingLLM (Xiao et al., 2024b), IM-infinite (Han et al., 2023), and MInference (Jiang et al., 2024), across the LongBench (Bai et al., 2023) and RULER (Hsieh et al., 2024) benchmarks. Our experiments are based on the I.a.M-2 7B model fine-tuned on se- quences of length 32K and 80K (Fu et al., 2024). For StreamingLLM (Xiao et al., 2024b), we use the official implementation, adjusting the attention sink to 4 and set- ting the attention context size to 2000. Similarly, for LM- infinite (Han et al., 2023), we follow the official code, con-
7

---

Core Context Aware Transformers for Long Context Language Modeling
Table Comparisons of different models multi-document EM score (Liu al., 2024b) evaluated lengths from 4K 128K. "FTT."
denotes the latency to generate the first token in the pre-filling stage. We report the latency within a context of 128K on two A800 GPUs. Methods
4K
8K
16K
32K
64K
128K
Avg.
FTI. (s)
LLaMA2- 7B-80K (Vanilla Self-Attention
39,4
37.8
37.6
36.2
34.6
30.3
36.0
124.85
StreamingLLM LLM (Xiao et al., 2024b)
33.6
26.0
32.2
30.6
27.4
25.1
29.2
34.74 (3,6x)
LM-Infinite (Han et al., 2023)
31.6
25.6
32.4
32.2
28.2
26.3
29.4
32.57 (3.8x)
Minference (Jiang et al., 2024)
39.0
32.4
37.4
36.0
32.3
28.9
34.3
20.18 (6.2x)
CCA-LLM (Ours)
39.3
33.2
35.4
31.4
35.3
32.0
34.4
15.89 (7.9x)
for short sequence modeling. For example, our CCA-LLM outperforms StreamLLM (39.3 vs. 33.6) and MInference (39.3 vs. 39.0) in terms of EM score. The reason is that our CCA-Attention captures context dependencies without discarding crucial tokens. In contrast, StreamingLLM (Xiao et al., 2024b) prioritizes attention on the initial and final to- kens, effectively discarding intermediate tokens, which may contain essential information. Similarly, MInference (Jiang et al., 2024) employs predefined sparse attention patterns, selectively attending tokens and potentially overlooking critical parts of the input sequence. Both approaches risk losing important contextual information, leading to subopti- mal performance in tasks requiring comprehensive under- standing. Our method, by preserving both local and global contexts, ensures that no critical information is overlooked, thereby achieving superior performance.
22.5
Longbench-Ei Score
22.0
21.5
LLaMA2-7B-80K
21.0
MInference
CCA-LLM (Ours)
20.5
3
5
6
9
Latency
Figure 3. Illustration of inference flexibility by adjusting the group size g and local window size to generate various CCA-LLM models with different latency and accuracies in the test time. This architectural flexibility allows for precise control over the trade-off between inference latency and accuracy, particularly beneficial for real-world applications with varying user traffic patterns.
For the extremely long context (e.g., 64K and 128K), Our CCA-LLM shows much better performance than vanilla self-attention in terms of EM score (35.3 vs. 34.6) and 7.9x inference speedup with a context length of 128K. The advan- tages of our method become more prominent as the length of the context increases, while the performance of vanilla self-attention may even decrease when the context length becomes very large. The reason is that in an extremely long context, non-core contexts (i.e., the irrelevant context) will be compressed by the proposed weighted pooling. In this way, CCA-LLM alleviates the redundant context issue and improves the long-context modeling performance.
and KV cache usage among all compared methods. ur CCA- LLM is able to reduce the KV cache usage while the best counterpart MInferencce only accerlerate the prefilling stage and does not reduce KV cache storage.
To further validate the effectiveness of our method across different model architectures, we conduct extensive experi- ments on more recent foundation models: LLaMA3.1 1-8B- Instruct-128K and (0wen2.5-7B-1258 As shown Table 2. our CCA-Attention demonstrates superior performance over the state-of-the-arr baseline MInference across three key metrics: computational efficiency, memory reduction, and model performance. The consistent improvements across different model architectures suggest that our approach is not limited to specific model designs.
4.3. Demonstration of Inference Flexibility
In real-world applications, user traffic exhibits diurnal vari- ations. During peak traffic periods, the full self- afffattention struggles with throughput limitations, necessitating addi- tional servers to accommodate the increased demand. In- stead, our CCA-LLM can dynamically adjust the group size g and the local window size during inference to improve throughput. During off-peat traffic periods, our CCA-LLM is able to enhance model performance, albeit with minor compromise in throughput. This strategy exhibits remark- able elasticity to address the variable user traffic challenges.
Comparisons on Long-document QA. We evaluate the performance of our CCA-Attention against other meth- ods, including StreamingLLM (Xiao et al., 2024b), LM- Infinite (Han et al., 2023), and MInference (Jiang et al., 2024), LLaMA2-7B-800 models using the multi- document EM Score metric. As shown in Table 3, we conduct comparisons across a range of context lengths: 4K, 8K, 16K, 32K, 64K, and 128K. For the short context (e.g., 4K). our CCA-LLM consistently achieves the highest EM score across all methods, showeasing significant capability
To verify this, we conduct experiments with different group sizes y and local window sizes during inference. In Fig- ure 3, our CCA-LLM demonstrates an improved Longbench-
8

---

Core Context Aware Transformers for Long Context Language Modeling
LLaMA2-7B-80K (Vanilla Self-Attentio))
MInference
LM-Infinite
CCA-LLM (Ours)
125
600
Memory Footprint (GB)
60
100
Latency (ms)
Lateney
75
400
40
50
20
25
4K
8K
16K
32K
64K
128K
4K
8K
16K
32K
64K
128K
4K
SK
16K
32K
64K
128K
(a) Pre-filling Lateney (FTI.
(b) Decoding Latency (TTL.
(c) KV
Cache Memory Usage
Figure 4. Comparisons with state-of-the tfrt methods in terms of both computational and storage overhead LLaMA2-7B "FTL" (first token latency) is the time taken to generate the first token after receiving the input in the pre-filling stage. "ITL" (inter token latency) is the time delay between generating consecutive tokens (except for the first token) during the decoding stage.
E Score with concomitant increase in computational over- head as g decreases. With a reduction in the group size g from 16 to 2 and the local window size s from 4096 to 1024, the Longbench-E Score escalates from 20.54 22.69, while the latency rises from 2.80 seconds 3.89 seconds. Despite training CCA-LLM only once, we obtain spectrum of models, each with distinct performance and computational demands. This flexibility comes from two complementary modules: The globality-aware pooling module dynamically compress core tokens based on semantic relevance via intra- group attention, enabling adaptation to different group sizes. The locality-preserving module uses rotary embeddings to encode relative positions, achieving translation invariance and maintaining local context across scales.
5. Conclusion
In this paper, we proposed a Core Context Aware Attention (CCA-Attentions for long-context language modeling with reduced computational overhead compared with vanilla self- attention. Our "CA-Attention includes two components: 1) globality-aware pooling module exploits the importance of imput tokens to encapsulates core tokens and employs them for attention, capturing global coarse- bearse-rrained information; 2) The lorality- module focuses on neighboring tokens to capture local fmed-grained context, serving as complement for the global module. Our proposed attention is able to replace the full self-attention in existing LLMs with a minimal finetuning efforts. Experimental results show the effectiveness of our CCA-Attention with promising per- formance and decreased computational cost.
4.4. Comparisons on Computational Efficiency
We compare CCA-LLM with LLaMA2-7B-80X equipped with full self-attention, LM-Infinite (Han et al., 2023), and MInference (Jiang et al., 2024) in terms of inference latency and memory footprint during forward- propagation on a single NVIDIA A800 GPU. The effi- ciency performance was assessed across a range of input sequence lengths, i.e., {4K, 8K, 16K, 32K, 64K, 128K). In Figure 4, our CCA-Attention achieves a 7.9x inference speedup than LLaMA2-7B-800 with the full self-attention (15.89s vs. 124.85s in 128K context) MInference (15.89s vs. 44.50s in 128K context) in the pre-filling stage. Our CCA-Attention also exhibits a reduced KV cache memory usage than ..aMMA-7B--88 (4.5GB vs. 64GB 128K context) and MInference (4.5GB vs. 64GB in 128K context). Note that Minference (Chen al., 2024) only accelerates the pre-filling stage and adopts full self-attention in the decoding stage. Moreover, it does not reduce KV cache, leading to the same memory usage as the full self-attention. Conversely, our CCA-LLM is able to accelerate both pre- filling and decoding stages with reduced KV cache, which
Acknowledgments
This work was partially supported by the Joint Funds of the National Natural Science Foundation of China (Grant No. No.U24420327), Key-Area Research and Development Pro- gram Guangdong Province 188000017000 the Major Key Project of Peng Cheng Laboratory (PCL) PCL2023A08, Postdoctoral Fellowship Program of CPSF (GZC2025104)) and TCL Science and Technology Innovation Fund, China
Impact Statement
Our work addresses the critical challenge of computational inefficiency in long-context modeling for large language models. By introducing a core context aware attention mech- anism, we achieve a reduction in attention complexity to linear time while preserving essential semantic interactions. This enables LLMs to process contexts up to 128K tokens with a 7.9x speedup over full self-attention, without com- promising accuracy. This promotes more environmentally friendly and sustainable approach to AI development, align- ing with the growing demand for more energy-efficient AI.
more practical in wolld applications.

---

Core Context Aware Transformers for Long Context Language Modeling
References
Dao, T. Flashattention-22: Faster attention with better paral- lelism and work partitioning. In International Conference Learning Representations, 2024.
AI, M. Introducing meta Ilama 3: The most capable ppenly available Ilm to date. https: / / //taammaaa...//aaaaaaaa......a.. meta. com/ 20.
2024. Accessed: 2025-01-
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-effiiien exact attention with io-awareness. In Advances in Neural Information Pro- cessing Systems, 2022.
Azerbayev, Z., Ayers, E., and Piotrowski, B. Proof- pile. In Available online: https//giihub.comthnngii azerbayev/proof-pil.. 2022.
Ding, J., Ma, S., Dong, I.., Zhang, X., Huang, S., Wang, W., Zheng. N., and Wei, F. Longnet: Scaling transformers to 1. 1,000,000,000 tokens. arXiv preprini arXiv:2307.02486. 2023.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J. and Li, Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng, H. Data engineering for scaling language models to 128k context, arXiv preprint 24X1v22402.10171, 2024.
Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.00 05500, 2020.
Guo, D., Yang, D., Zhang, H.,, Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 122428281..28,, 2025.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu,
Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang. S. LM-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308. 2023.
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray. S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding, In International Conference on Learning Representations, 2021.
Cerebras. Slimpajama: A 627b token cleaned and deduplicated version of redpajama. https:/ /www. cerebras net, 2024. Accessed: 2025-01-20.
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404. 06654, 2024.
Chang, Y., Wang. X., Wang, J., Wu, Y., Yang. L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang. Y., Yu, S., Yang, Q., and Xie, X. A survey on evaluation of large language models. ACM Transactiony Intelligent Systems and Technology. 15(3):39:1-39:45 2024.
Jiang, H., LI, Y., Zhang, C., Wu, Q. Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long- context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, 2024.
Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 20..23666.5999 15595, 2023.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are mns:: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165. PMLR, 2020.
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. LongLoRA: Efficient fine-tuning of long- context large language models. International Conference on Learning Representations, 2024.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Belanger, D., Colwell, L., et al. Masked language modeling for proteins via linearly scalable long contexttext transformers. arXiv preprint arXiv:2006. 200.22000.33555 2020.
Liu, A., Feng. B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint 2412 2024a.
10

---

Core Context Aware Transformers for Long Context Language Modeling
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, E., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.
Tillet, P., Kung, H., and Cox, D. D. Triton: an intermediate language and compiler for tiled neural network compu- tations. In Proceedings of the International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Manakul, P. and Gales, M. F. Long-span summarization via local attention and content selection. In Zong, C., Xia, F., Li, W. and Navigli, R. (eds.), Association for Computational Linguistics, pp. 1026-60411 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Mohtashami, A. and Jaggi, M. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36:54567-54585, 2023.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., Bikel, D., Blecher, L.,, Canton- Ferrer, C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, Fu, W., Fuller, B., Gao, C.,, Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams, A., J. X., Xu, P., Yan, Z., Zarov, L., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 2023b.
Mu, J., Li, X., and Goodman, N. D. Learning to com- press prompts with gist tokens. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,
LA, USA, December 10- 16, 2023, 2023.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Ouyang, L., Wu, J.. Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.., Simens, M., Askell, A., Welinder, P., Christiano. P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Alten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 1998-6008, 2017.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Efficient context window extension of large language models. In International Conference on Learning Repre- sentations, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M.. Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. -hain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, pp. 24824-24837, 2022.
Qin, G. and Van Durme, B. Nugget: Neural agglomera- tive embeddings of text. In International Conference on Machine Learning, pp.
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. InfLLM: Training-free long- long-context extrapolation for lims with an efficient context memory. In Advances in Neural Information Processing Systems, 2024a.
PMLR, 2023.
Qin, G., Rosset, C., Chau, E., Rao, N., and Van Durme, B. Dodo: Dynamic contextual compression for decoder- only lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers). pp. 9961-9975, 2024.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks. In International Conference on Learning Representations, 2024b.
Rae, J. W., Potapenko. A., Jayakumar, S. M., Hillier, C.,, and Lillicrap, T. P. Compressive transformers for long- range sequence modelling. In International Conference on Learning Representations, 2020.
Xiong, W., Liu, J., Molybog, I,, Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang. H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M.. Wang, S., and Ma, H. Effective long-context scaling of foundation models. In Proceedings of the Conference
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.0862/ 2023.

---

Core Context Aware Transformers for Long Context Language Modeling
of the North American Chapter of the Association for Computational Linguistics, pp. 4643 4663, 2024.
Yang. A., Yang, B., Zhang, B., Hui. B., Zheng, B., Yu,
B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.55115, 2024.
Yang, B., Wang, L., Wong, D. F., Shi, S., and Tu, Z. Context- aware self-attention networks for natural language pro- cessing. Neurocomputing, 458:157-169, 2021.
7aheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer se- quences. Advances in Neural Information Processing Systems, 13:1733117 2020.
Zhang, P., Liu, Z., Xiao. S., Shao, N., Ye, Q. and Dou, Z. Long context compression with activation beacon. International Conference on Learning Representations, 2024.
12

---

Core Context Aware Transformers for Long Context Language Modeling
SUPPLEMENTARY MATERIALS
Contents
Theoretical Analysis Reachability for CCA-Attention
14
More Implementation Details
15
B.1 More Details on Dataset Evaluation Metries
15
B.2 More Experimental Protocols
15
More Experimental Results
17
C.I Experiments on More Long Context Benchmarks
17
C.2 Comparisons with More Efficient Attention Methods
17
C,3 More Comparisons on MMLU with Multi-choice QA
17
C.4 Statistical Results of Sparse Attention Scores
17
C.5 More Ablation Studies
18
C.6 Training Convergence Curve
19
13

---

Core Context Aware Transformers for Long Context Language Modeling
A. Theoretical Analysis on Reachability for CCA-Attention
In the self-attention, the calculation can be formulated as Attention Q K, V) softmax QK QQT/VV)) V, where Q=Xwº, K=Xw*, and V=XWV, Wº, W WV are learable parameters. For convenience in analyzing the attention mechanism, we denote the attention weight as A = softmax (QK /vd), where the element in represented as aij. We give the definition of reachability, and show that our method is satisfy the information among all tokens is reachability, and then give the concrete expression of attention output.
Definition 1. (Reachability) We say the token is reachable from the token in the attention map if and only the attention weight from the token j is positive, i.e., Oij 0.
Proposition The attention score with causal masking the CCA-Attention mechanism fully satisfies the reachability condition from the earlier tokens the later tokens in the sequence each transformer layer. Moreover, the final oulput representation R for i-th token in Eqn. (5) can be given by
V 9:1r 5. Pp
iSi::
(6)
Eq=
1 p.
V
E V migttj w=s+(i s) mod g: clse,
p=1
where oj is the j-th element of the output o, & is the weight of all core tokens in Eqn. (2), A denote the attention score of CCA-Attention in Eqn. (5).
Proof. We decompose the i-th row of the attention scores A into two parts:
A
A A
where A R and A R" with w=s+(i- s) mod g. We aim these two terms to formalize A, element by element into the structure of full attention. For simplicity, we expand each element in these two terms with the weight + although the is not directly weighted on the attention:
A =
11 11 1 A ......,
A T Pm AG ......, o Pm kmm
A. A.......
A iil,...,, L.
(8)
Based on the property of the attention scores of A for Vi > j. we have ai > 0, satisfying the condition of reachability. Next, we drive the output representation o of a token.
When S g. for each oj in o, we can use both the attention score the locality-preserving module and the globality-aware pooling module obtain
Oj
Σ A 1g
V
(9)
q=1
When > g. for each 0j in o, the attention of the secaity-presserve module obtain
oj
+
A V mggttj
(10)
9=1
Taking Eqn. (8) and Eqn. (10) together, we obtain the results.
14

---

Core Context Aware Transformers for Long Context Language Modeling
B. More Implementation Details
B.I. More Details on Dataset and Evaluation Metries
SlimPajama (Cerebras, 2024) dataset is an open-source reproduction of the data mixture used to pretrain the LLaMA models. It consists of 82% web data, 4.5% code from Github, 4.5% Wikipedia, 4.5% books, 2.5% Arxiv, and 2.0% StackExchange, used for extending the context lengths of LLMs 128K tokens through careful data engineering techniques like per-source length upsampling. We use the SlimPajama dataset (Cerebras, 2024) as our training dataset.
LongBench (Bai et al., 2023) is a pioneering benchmark designed for the bilingual, multitask, and comprehensive assessment of the long context understanding capabilities within LLMs. It encompasses diverse languages, specifically Chinese and English, thereby facilitating a more exhaustive evaluation of the multilingual proficiencies of large models in long context scenarios. Moreover, LongBench is structured with 6 major categories and 21 distinct tasks, spanning crucial long-text application areas such as single- single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. LongBench has 14 English tasks, Chinese tasks, and 2 code tasks. The average length of the majority of tasks falls within the range of 5k to 15k, and it comprises total of 4,750 test data. For detailed statistical information and construction methodologies of LongBench tasks, reference can be made to the designated source. Additionally, LongBench-E is a test set featuring a more evenly distributed length constructed through uniform sampling. It contains comparable data quantities in the 0-4K, 4K-8K, and 8K+ length intervals, enabling an in-depth analysis of the model's ' performance fluctuations across different input lengths. We conduct the experiments on LongBench-E to verify the long context understanding capability of models in Section 4.2.
Exact Match Score (EM Score) (Liu et al., 2024b) measures the model's ability find the key information within long context in multi-document question-answerte documents to reach the specified context length (20 for 4K, 48 for 8K, 96 for 16K, 190 for 32K, 378 for 64K, 755 for 128K), followed by a question. We evaluate EM score metric with the multi-document question-answeringg dataset in Lost in the Middle (Liu et al., 2024b), which is collected from Natural Queuuaiiuesinnos-ooo Open and Wikipedia. We use the exact match score as the evaluation metric. judging whether any of the correct answers appear in the predicted output in Section 4.2.
task. In this task, each test sample comprises certain number of
Massive Multitask Language Understanding (MMLU) (Hendryeks et al., 2021) dataset is designed to assess the capabilities of language models across a wide array of subjects, delving deeper into their academic and professional understanding. The MMLU benchmark spans 57 diverse subjects, ranging from elementary mathematics to professional law. The questions are designed to test both world knowledge and problem-solving abilities, challenging models with content from elementary to advanced professional levels. We use the MMLU metric to evaluate the model's 's proficiency across a diverse set of imguage-understanding tasks. It tests the model's ability to apply its knowledge to a broad spectrum of topics and question types, reflecting its generalization capability in real-world scenarios. The MMLU metric (Hendrycks et al., 2021), which tests world knowledge and gobblem-ollvigg abilities in zero-shot and few-shot settings, is evaluated using the MMLU dataset (Hendrycks et al., 2021). This dataset spans 57 subjects across disciplines such as STEM, humanities, and social sciences. We test the MMLU metric in 5-shot setting with MMLU dataset (Hendrycks et al., 202 to verify the commonsense generalization ability of models in the supplementary materials C.3.
Perplexity (PPL. quantifies how effectively a model can predict the context. It is calculated as the exponentiated average negative log-likelihood of a sequence, offering a statistical measure of language modeling performance. Proof- pile (Azerbayev et al., 2022) a 13GB high-quality dataset of mathematical text and code that comprises 8.3 billion tokens (measured by the gpt-neox tokenizer). The dataset is composed of diverse sources of both informal and formal mathematics and the raw data downloaded from the web. We report PPL on the test dataset. We use the test dataset of Proof-pile to verify long-context language modeling ability of models in the supplementary materials C.5.
B.2. More Experimental Protocols
CCA-Attention (Ours). For the continuous pretraining, we adopt the SlimPajama (Cerebras, 2024) dataset, an open-source replication of the LLaMA pretraining data mixture. This dataset comprises 82% web data, split between 67% from CommonCrawl and 15% from C4, alongside 4.5% from GitHub code, 4.5% from Wikipedia, 4.5% from books, 2.5% from Arxiv, and 2.0% from Stack Exchange. We replace the full self-attention in LLaMA2 with our proposed CCA-Attention. The number of groups in globality-aware attention is shared across different model sizes. Training is conducted on 8 x A800 GPUs using a micro- micho-batch size of and a gradient accumulation of 8, with a total of 1000 training steps. This training
15

---

Core Context Aware Transformers for Long Context Language Modeling
configuration is applicable to all model sizes and context lengths. Our method requires finetuning on modest number of tokens to extend the long-context capabilities of LLMss, enabling efficient attention computation. Specifically, require only 2.10 billion tokens for 32K and 5 billion tokens for 80K, which is significantly lower than the token requirements for retraining large language model.
To scale the models to long contexts, we modified the "base frequency" in RoPE from 10,000 to 500,000, following (Cerebras, 2024; Xiong al., 2024). the globality-aware attention, we set the position embedding of Kalobal the position embedding of the token at the middle position in the corresponding group, ensuring that our attention maintains positional awareness. Following FlashAttention (Dao, 2024), we implement our CCA-Attention by leveraging Triton (Tillet et al., 2019) perform low-level operator fusion between our globality pooling and locality-presert modules. This enables us to integrate our CCA-Attention standalone, cache-friendly operator, effectively eliminating redundant computations.
Compared Methods. We conduct comprehensive comparisons between our proposed CCA-Attention and several state- of-the-art methods, including LLaMA-2 with vanilla attention, StreamingLLM (Xiao et al., 2024b), LM-infinite (Han al., 2023), and MInference (Jiang al., 2024), across the LongBench (Bai al., 2023) and RULER (Hsich al., 2024) benchmarks. Our experiments based on the LLaMA-2 7B model fine-tuned on sequences of length 32K and 80K (Fu al., 2024).
For StreamingLLM (Xiao et al., 2024b), we use the official implementation, adjusting the attention sink to 4 and setting the attention context size to 2000. Similarly, for LM-infinite (Han et al., 2023), we follow the official code, configuring the local branch size to 1024 and the global branch size to 16. In the case of MInference (Jiang et al., 2024), we also employ the official code implementations, configured with the official settings.
16

---

Core Context Aware Transformers for Long Context Language Modeling
C. More Experimental Results
C.1. Experiments on More Long Context Benchmarks
We further evaluated the long-context modeling performance of our proposed method on RULER (Hsich et al., 2024) using the LLaMA2-7B-800 model across various context lengths (ranging from 8K 64K). As shown Table 3, our approach consistently outperforms MInference (Jiang al., 2024) all context lengths, demonstrating its superiority in context modeling. The performance gains primarily stem from two key innovations in method: 1) Globality-awand Pooling Module dynamiclly identifies and pools the task-relevant context into core tokens, which effectively reduces redundancy while preserving essential information; 2) A comprehensive information interaction across all context segments, as opposed to simply discarding tokens.
module that supplements local information and ensures
Table 4. Comparisons on RULER across 8-64K context.
Methods
8K
16K
32K
64K
Avg.1
LLaMA2-7B-80K (Vanilla Self-Attention)
71.90
66.26
61.54
55.15
63.71
MInference (Jiang al., 2024)
67.78
65.32
61.43
52.77
61.83
CCA-LLM (Ours)
68.15
66.31
60.89
54.88
62.56
C.2. Comparisons with More Efficient Attention Methods
To further evaluate method, we compare it with LongLoRA (Chen al., 2024), recently proposed training- based approach with PI techniques. As shown in Table 5, on LongBench-E, our CCA-LLM achieves superior performance in terms of modeling accuracy, inference speed, and memory efficiency. Notably, while 52-Attention proposed by LongLoRA is only available during training, it defaults to full self-attention during inference, resulting in comparable computational and memory overhead to standard attention mechanisms. contrast, CCA-LLM achieves a first-token latency reduction of 3.5x and 46% decrease in memory usage, demonstrating its effectiveness for efficient long-context modeling.
Table 5. Comparisons on LongBench-Fi "FTI." denotes the latency to generate first token.
Model
Avg. Score
FTL (s)
Memory. & (GB)
LLaMA2-7B-328
22.11
9.15
35.58
LongLoRA
21.58
9.15
35.58 (0%1)
CCA-LLM (Ours)
21.86
2.59
19.12 (46%))
C.3. More Comparisons on MMLU with Multi-choice QA
When applied to tasks with a fixed input length, such as multi-choice QA tasks, we set the number of groups m constant value. This ensures that the overall computational complexity of our method is O(L), where L represents the input sequence length. this section, we compare the performance of our method with full self-attention on the MMLU dataset specifically for multi-choice QA tasks. As shown in Table 6, our method consistently achieves superior performance than the traditional aeffaattenion method, demonstrating the effectiveness of our approach.
Table Comparisons on MMLU with multi-choice QA.
LlaMA2-7B-8K
LlaMA2-7B-16K
LlaMA2-13B-16K
LlaMA2-13B-32K
Method
Full Self-attention
Ours
Full Self-attention
Ours
Full Self-attention
Ours
Full Self-attention
Ours
MMLU
33.34
37.55
28.19
39.71
27.17
48.11
26.72
47.93
C.4. Statistical Results of Sparse Attention Scores
We visualized LLaMA2-7B' attention scores on a sentence of 32 tokens in Figure as a supplement to Figure 1. As shown in the figure, these attention scores show consistent sparsity from shallow to deep layers. Same as demonstrated in existing methods (Beltagy et al., 2020; Xiao et al., 2024b).
17

---

Core Context Aware Transformers for Long Context Language Modeling
Layer Head
Layer
Layer Head 9
Layer 8 Head
1.0
1.0
1.0
1.0
12
0.6
12
066
0.6
0.6
16
16
16
16
20
0.4
0.4
0.4
0.4
24
24
24
0.2
022
8
12
16
20
24
28
0
12
16
20
24
28
0
8
12
16
20
24
28
0
8
12
16
20
24
28
Layer 12 Head 9
Layer 18 Head
Layer 24 Head
Layer 30 Head
0.8
0.8
08
0.8
12
0.6
12
0.6
12
0.6
12
0.6
16
20
20
20
20
0.2 28
0.2
02
0.2
26
28
26
24
16
24
16
24
Figure 5. A visualization of attention scores in 1.L.MA27BB with a sentence of 32 input tokens. The attention map reveals a distinct pattern: the majority of tokens exhibit minimal attention scores. Conversely, minority of tokens are associated with significantly higher attention scores. This trend is observed consistently from the shallow to the deeper layers of the model.
Based on these observations, our CCA-Attention of assessing token importance within each group using the attention from the last token both rational and effective. The attention map visualization reveals distinet pattern where tokens that are important to the query receive consistently high attention scores from all subsequent tokens. This indicates that important tokens, regardless of their position within group. have notable influence attention distribution, suggesting that our method of importance assessment is capable of capturing these crucial tokens.
C.5. More Ablation Studies
Effect of Group-wise Pooling Strategy. For computational efficiency, conduct ablations with LLaMA2-7B-16K We adopt perplexity (PPL.) to evaluate our CCA-Attention models. To investigate the effect of different pooling strategies, we conduct ablations with max pooling. mean pooling and our weighted pooing in Eqn. (2). In Table 7, our CCA-Attention with group-wise attention pooling strategy achieves superior results in both PPL (e.g., 2.85 vs. 2.99). This advantage arises since max pooling retains only the token with the highest response, thereby discarding the semantic importance of the remaining tokens. Mean pooling averages all tokens within a group, which substantially dilutes the semantic significance of critical tokens. In contrast, our CCA-Attention dynamically assigns aggregation weights of each token, facilitating more comprehensive and efficient fusion.
Table 7. Effect of pooling strategy.
Strategy PPL.
Mean Pooling
Max Pooling CCA-Attention (Ours)
2.99
2.99
2.85
Effect of Group Size g. To investigate the effect of different group sizes g, we implement the proposed CCA-Attention with different y E (2, 4,8, 8, 16, 32, 64). Table 8, as y increases, the computational efficiency improves while the PPL. increases. Upon closer examination, the smallest group size g captures the most comprehensive information, which translates to the highest computational cost but also the optimal PPL. Conversely, an excessively large g leads overemphasis globality-aware attention, compressing information to the point where crucial semantic nuances may be overlooked, thereby curtailing performance. To strike balance between computational efficiency and model performance, we have selected g-16 as the default training setting.
Effect of Local Window Size s. To systematically evaluate the influence of different local window sizes w, we implement the proposed CCA-Attention across a range of E (256, 512, 1024, 2048, 4096). Table an increase correlates with
18

---

Core Context Aware Transformers for Long Context Language Modeling
Table 8. Effect of group size g.
2
8
16
32
64
PPL. +
2.75
2.78
2.81
2.86
2.90
2.93
Latency & (ms)
546.7
503.5
479.6
462.8
457.74
456.0
lower PPI., but this is counterbalanced by rise in computational cost. A larger captures contextual information with neighborhood tokens, but also increases computational demands. Conversely, smaller indicative of limited receptive field, constrains the exchange of information within the Striking a balance between computational efficiency and model efficacy, we opted for s=1024 as the default training setting.
module, resulting in diminished performance.
Table 9. Effect of local window size 8.
256
512
1024
2048
4096
PPL L
2.98
2.92
2.86
2.79
2.73
Latency (ms)
457.4
460.1
461.4
462.8
473.1
Effect of Different Updating Strategies. As mentioned in Section 3.4, we have two updating strategies: 1) updating all the parameters during finetuning (full finetuning) and 2) only updating parameters WΩ, W WV (partial finetuning). In Table 10, we compare these two variants of our methods on the Longbench-F benchmark. The variant that updates all parameters during finetuning achieves better performance because allows the model fully adapt to our proposed attention. In contrast, partial fine-tuning, while limiting the model's adaptability due to fixed pre- pre-trained features, still achieves competitive performance. This makes partial fine-tuning practical choice in scenarios requiring rapid training or where computational resources are limited. Despite its constraints, partial fine-tuning can deliver performance close to that of full fine-tuning, offering a balanced trade-off between efficiency and accuracy.
Table 10. Effect of different updating strategies.
Strategies
Single- Doc. QA Multi-Doc. QA
Sum.
FS. Learning
Synthetic
Code
Avg.
Purtial Finetuning
5.39
3.62
9.21
60.41
1.34
51.77
21.96
Full Finetuning
5.62
4.34
8.99
59.60
0.48
54.40
22.24
C.6. Training Convergence Curve
In the experiments, finetune the L.LMMA-7B-322 and LLaMMA27B-800 models equipped with our CCA-Attention SlimPajama (Cerebras, 2024) dataset for 1,000 iterations. We show the training convergence curves of both models with CCA-Attention in Figure 6. From the results, by minimizing the training loss, both LLaMA2-7B-32K and LLaMA2-7B-88 models are able to converge very fast. The perplexity rapidly converges within approximately the first 100 iterations and remains stable over 1,000 iterations. These results not only demonstrate the effectiveness and training stability of our proposed CCA-Attention, but also establish has the potential to be a plug-and-play attention module incorporated into existing LLMs. Notably, initial training loss of LLaMA2-7B-322 is higher than that of LLaMA2-7B-800 This difference arises because LLaMA2-7B-328
finetuned from the official LLaMA2-7B-4K model, which has shorter context window and thus requires more significant adjustments adapt longer sequences. In contrast, LLaMA2-7B-s6 --KKKK is fine-tuned from a model pre-trained by Fu, etal. (Fu et al., 2024).
19

---

Core Context Aware Transformers for Long Context Language Modeling
10a.
10a
CCA-LLaMA-2 7B (32K Context)
CCA-LLaMA-2 7B (80K Context)
Perplexity
Perplexity
103
101.
10
200
400
600
800
1000
200
400
600
800
1000
Training Itertation
Training Itertation
Figure Convergence curves of our CCA-LLM models under different contexts.
20


Reviews:
[{"review_id": "6KTK6oOgUx", "reviewer": "Reviewer_qgiU", "summary": "This paper addresses the computational inefficiency and redundancy in Transformer-based Large Language Models (LLMs) when processing extremely long contexts (e.g., 128K tokens). The authors propose Core Context Aware (CCA) Attention, a plug-and-play mechanism comprising two modules: (1) a globality-aware pooling module that dynamically compresses input tokens into core tokens via weighted grouping, and (2) a locality-preserving module that retains fine-grained local context. The fusion of these modules reduces computational complexity from quadratic to linear while preserving full token reachability. Experiments on LongBench and multi-document QA benchmarks demonstrate superior performance (e.g., 7.9× speedup at 128K context) over methods like MInference and StreamingLLM, with reduced memory footprint.\n\n## update after rebuttal\nThanks to the authors for addressing my concerns and providing additional results. \nI didn't find novelty issue on the idea of core context modeling on my side.\nI will keep my score.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1. Figure 3 demonstrates stable performance with varying group sizes (g) and window lengths (s) during inference, what are the possible reasons?\n\nA1. We deeply appreciate your valuable feedback. We design our CCA-Attention for flexibility, enabling promising performance across varying $g$ and $s$ settings without requiring additional fine-tuning. The reasons are twofold:\n\n- Globality-aware pooling module dynamically **pools groups of tokens into core tokens based on their importance, regardless of the group size $g$**. The pooling weights are derived from intra-group attention scores, which depend on semantic relevance rather than fixed positional patterns. As a result, the module generalizes to arbitrary group sizes $g$ during inference.\n - Locality-preserving module is inherently translation-invariant to window boundaries by employing rotary position embeddings that encode relative distances rather than absolute positions. This relative distance encoding naturally **decouples the module's operation from specific window size $s$**, allowing adaptation to different $s$ in inference while preserving local context relationships.\n\nWe would include these discussions in the revised paper.\n\n> Q2. Could the core token (Eqn.2) be biased toward later tokens in a group due to using the last token’s query?\n\n\nA2. In our CCA-Attention, the strategy of deriving core tokens based on the attention scores from the last token is both rational and effective. This is supported by two points:\n\n - The attention map visualization in Section C.2 of supplementary material reveals a distinct pattern: **tokens that are important to the query receive consistently high attention scores from all subsequent tokens**. This indicates that important tokens, regardless of their position within a group, have a notable influence on the attention distribution. This suggests that our pooling weights assessed by the last group token are able to capture these crucial tokens.\n - The consistent high performance in long-context modeling tasks confirms that our CCA-Attention effectively captures global and local dependencies within long contexts. This is a direct result of our pooling strategy, which shows that information relevant to the query is not overlooked.\n\n\nWe would include these discussions in the revised paper.\n\n> Q3. Without requiring additional experiments, could the authors discuss: is there a principled way to set m for unseen tasks?\n\nA3. We appreciate the reviewer's insightful comments. In practice, we are able to **set different group sizes according to the nature of the task**. For high-precision tasks (e.g., legal document analysis), smaller group sizes are preferable to ensure finer granularity and capture more detailed contextual dependencies. Conversely, for tasks prioritizing efficiency over precision (e.g., real-time dialogue systems), larger group sizes are able to reduce computational overhead. This trade-off aligns with the inherent flexibility of our proposed CCA-Attention, which allows dynamic adjustment of m during inference.\n\n\n> Q4. Could the authors explain why the model performance does not consistently degrade with increasing text length in Table 2?\n\nA4. As discussed in the paper, we discovered that full self-attention may face redundant context issues in sequence modeling. **This redundancy may hamper the modeling performance** by 1) weakening the focus on semantically important content, and 2) introducing noise from irrelevant context. More importantly, **this issue becomes more severe with longer sequences**.\n\nTo address this, we propose a core context aware attention mechanism, in which **non-core/irrelevant contexts will be compressed by weighted pooling**. In this way, CCA-Attention not only alleviates the redundant context issue but thus improves the long-context modeling performance. As a result, the advantages of our method become more prominent as the length of the context increases.\n\n> Q5. Will the assumption of redundant text be related to the topic of testing cases? Math and general QA may have different degrees of redundancy.\n\nA5. **Our assumption of redundant context is not related to the topic**. To verify this, we perform an analysis using LLaMA2-7B on 16K length corpora from two domains: math (MathPile dataset[1]) and general QA (WikiText dataset[2]).\n\nBy examining the attention weights of the final token relative to preceding tokens, we quantify the redundancy degree as the proportion of tokens receiving attention weights below 0.001. **The results demonstrate comparable levels of redundancy between math** (99.8% tokens' weights below 0.001) **and general QA context** (98.7% tokens' weights below 0.001), supporting our initial assumption.\n\n[r1] A Billion-Token-Scale Pretraining Corpus for Math. NeurIPS 2024.\n\n[r2] Pointer Sentinel Mixture Models. ICLR 2017.\n\n---\nWe sincerely hope our clarifications above have addressed your concerns.", "claims_and_evidence": "The claims in the paper are supported by clear evidence.", "methods_and_evaluation_criteria": "The methods presented in the paper are well-motivated, aiming to balance computational efficiency with model performance. However, I still have some question:\n\n1. Figure 3 demonstrates stable performance with varying group sizes (g) and window lengths (s), what are the possible reasons why changing the size of g and s can work also well during inference?\n2. Could the core token (Eqn.2) be biased toward later tokens in a group due to using the last token’s query?", "theoretical_claims": "I have thoroughly reviewed the theoretical claims in the manuscript, which are well-structured and insightful. The authors provide a rigorous reachability guarantee through Proposition 1, demonstrating that the CCA-Attention mechanism preserves full token accessibility by integrating global and local attention scores. This formalization ensures that all tokens remain interconnected, addressing a critical aspect of long-context modeling. Additionally, the computational and storage complexity analysis is comprehensive, highlighting the method's superiority in reducing computation costs and optimizing key-value cache storage. These analyses underscore the method's efficiency and effectiveness for handling long-context tasks.", "experimental_designs_or_analyses": "The experimental design is comprehensive, covering a range of benchmarks and metrics to validate the proposed method's effectiveness. The use of LongBench (Bai et al., 2023) and multi-document EM scores (Liu et al., 2024b) provides a robust assessment of long-context understanding. The inclusion of various model sizes (LLaMA2-7B-32K and LLaMA2-7B-80K) ensures scalability analysis. The comparison with state-of-the-art methods (StreamingLLM, LM-Infinite, MInference) highlights the competitive edge of the CCA-Attention mechanism.\n\nI have a few questions regarding the experimental design:\n1. The paper explores a fixed group size m for token grouping. While this simplifies experimentation, real-world applications often require balancing accuracy and efficiency across diverse tasks. Without requiring additional experiments, could the authors discuss: is there a principled way to set m for unseen tasks (e.g., smaller groups for high-precision tasks vs. larger groups for efficiency) ?\n2. Table 2 shows that model performance does not consistently degrade with increasing text length. A more in-depth analysis or explanation would enhance the understanding of the method.", "relation_to_broader_scientific_literature": "Although CCA builds on some sparse attention (eg., Longformer) and linear approximations (eg., RetNet), it introduces dynamic token compression, significantly reducing the computational costs and storage demands for the attention while outperforming other baseline methods like LM-Infinite and MInference.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n\n1. The method's primary strength lies in its compatibility with pretrained LLMs, such as LLaMA2. This plug-and-play design minimizes the need for extensive retraining, making it easier to integrate into existing models.\n2. CCA integrates Triton, whose ability to optimize low-level operations enhances the method's efficiency, enabling faster inference times.\n \nWeaknesses:\n1. The method currently uses a fixed group size for inference, which limits its adaptability to varying text lengths and complexities. The paper lacks an analysis of adaptive grouping strategies, which could improve the method's flexibility and performance.", "other_comments_or_suggestions": "1. The claim of “significant improvements” seems over-stated. Adding “compared with other baseline methods” would make it more accurate.\n2. Is it be clear if $C_t^1(x_t)$ is revised as $C^1(x_t)$?\n3. In Figure 2, the bold font usage of g is inconsistent, and $A^G$ is confusing to me.\n4. The meaning of “LLaMA2-7B-32K” and “LLaMA2-7B-80K” should be specified.\n5. The notations $K$ and $V$ in 12-th in Algorithm 1 should be performed by \\tilde.", "questions_for_authors": "1. will the assumption of redundant text be related to the topic of testing cases? Math and general qa may have different degree of redundancy", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4a7RCVoy6", "reviewer": "Reviewer_JpcZ", "summary": "This work proposes Core-Context-Aware attention to enhance efficient long-context modeling for LMs. Two modules are included: a global pooling module to compresses groups of tokens into core tokens and a local module to preserve local information. Experiments on various tasks and datasets show the effectiveness of the proposed approaches.\n\nUpdate after rebuttal:\n\nI do appreciate the authors' promise of improvements and extra results. Nevertheless, I think my major concern over the similarities between this work and previous work from the perspective of methodology is still not addressed, which is why I will keep my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The proposed methods share similarities to previous context compression methods[r1-r6]. r1: Compressive Transformers, r2: Gist, r3: NUGGET, r4: Landmark, r5: Beacon, r6: DODO.\n\nA1. Thanks for your valuable comments. We clarify the differences below:\n\n- **Problem Importance & Motivation**\nThe core challenge we tackle is efficient and effective long-context modeling in LLMs. While self-attention handles long sequences, its quadratic complexity and redundancy in ultra-long contexts (e.g., 128K) severely degrade efficiency and modeling performance. Unlike prior works either compress features via auxiliary networks [r1] or compress context with extra specific tokens [r2-r6]), our method aims to dynamically identify and strengthen task-relevant core-context while suppressing redundancy. This distinction is crucial: compression prioritizes shortening sequences, whereas our core-context awareness optimizes context redundancy in self-attention computation for long context modeling.\n- **Novelty & Differences**\nOur CCA Attention introduces two innovative, complementary mechanisms:\n  - Globality-aware Pooling Module: Unlike context compression methods [r1-r6], it dynamically groups tokens and pools them into core tokens based on their importance. This ensures the model **focuses on task-relevant context while reducing redundancy**. \n  - Locality-preserving Module: While prior compression methods may have a risk of losing local details, our module selectively preserves neighboring tokens to maintain locally relevant information.\n\n\n**We would cite the above papers and include these discussions in the revised manuscript.**\n\n---\n\n> Q2. It might be interesting to explore more long-context tasks in addition to LongBench, such as RULER.\n\nA2. According to your suggestions, we conduct more experiments on RULER on LLaMA2-7B-80K with varying context lengths (4K-64K). From the results in Table I, our method achieves **superior performance over MInference, a state-of-the-art counterpart**.\n\nTable I: Comparisons of LLaMA2-7B-80K on RULER.\n|Methods|8K|16K|32K|64K|Avg.↑|\n|-|-|-|-|-|-|\n|LLaMA2-80K|71.90|66.26|61.54|55.15|63.71|\n|• MInference|67.78|65.32|**61.43**|52.77|61.83|\n|• CCA-LLM|**68.15**|**66.31**|60.89|**54.88**|**62.56**|\n\n> Q3. It would be interesting to explore whether the proposed approach can help more recent models (such as LLama 3 and Qwen models).\n\nA3. We conduct more experiments on LLaMA3.1-8B and Qwen2.5-7B with context length 128K in Tables II and III, respectively. From the results, our method is able to help LLaMA3.1 and Qwen models improve computational efficiency and memory reduction in long-context modeling.\n- **Computational efficiency**: CCA-Attention achieves 3.1× on LLaMA3.1-8B, and 3.9× on Qwen2.5-7B with 32K context.\n- **Memory reduction**: CCA-Attention reduces 49% GPU memory footprint on LLaMA3.1-8B, and 45% on Qwen2.5-7B.\n- **Performance**: CCA-Attention consistently outperforms MInference across all models, maintaining competitive accuracy with valina self-attention.\n\nTable II. Comparisons of Llama3.1-8B-Instruct-128K on LongBench-E. ''FTL'' denotes the latency to generate the first token in the pre-filling stage.\n|Model| Avg.↑ (%)|FTL↓ (s)|Mem.↓ (GB)|\n|-|-|-|-|\n|Llama3.1-128K|37.93| 9.55 |40.38 |\n|• MInference|37.74|4.93 (1.9x)|35.95(11%↓)|\n|• **CCA-LLM (Ours)**|**37.81**|**3.08** (**3.1**x)|**20.63** (**49**%↓)|\n\nTable III. Comparisons of Qwen2.5-7B-128K on LongBench-E.\n|Model| Avg.↑ (%)|FTL↓ (s)|Mem.↓ (GB)|\n|-|-|-|-|\n|Qwen2.5-128K|38.38|10.58 |35.11 |\n|• MInference|36.72|4.86 (2.2x)|32.40 (8%↓)|\n|• **CCA-LLM (Ours)**| **38.08**|**2.74** (**3.9x**)|**19.31** (**45**%↓)|\n\n\n> Q4. There are some key hyper-parameters, such as the group size and local window size. It would be nice to include more analyses and ablation studies.\n\nA4. In **Section C.3 of supplementary**, we show ablation studies on group size $g$ and local window size $s$. From the results, as $g$ increases, the efficiency will improve and the performance will decrease (in terms of PPL); as $s$ increases, the performance will improve and the efficiency will reduce. To achieve a balance, we select $g=16$ and $s=1024$ as default training setting.\n\n> Q5. How much data will be needed to fine-tune the model?\n\nA5. In **Section B.2 of the supplementary**, we show the details for fine-tuning: We use SlimPajama dataset to fine-tune the model, with only 2.1 billion tokens for LLaMA2-7B-32K and 5.0 billion tokens for LLaMA2-7B-80K. \n\n---\n\nWe sincerely hope our clarifications above have addressed your questions.", "claims_and_evidence": "Mostly.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs seem reasonable to me.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "There have been a line of context-compression methods not fully discussed and compared:\n\nPooling-styled compression: \n- Compressive Transformer: https://arxiv.org/pdf/1911.05507\n\nSpecial-token based compression:\n- Gist: https://arxiv.org/pdf/2304.08467\n- Nugget: https://arxiv.org/pdf/2310.01732\n- Landmark: https://arxiv.org/pdf/2305.16300\n- Beacon: https://arxiv.org/pdf/2401.03462\n- Dodo: https://arxiv.org/pdf/2310.02409", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is overall easy to understand.\n- The proposed methods have been shown effective for various tasks and datasets.\n\nWeaknesses:\n- The proposed methods share a variety of similarites to previous directions of context compression, some of which are not well discussed and compred with.\n- There is a lack of ablation studies and analyses on some design factors of the proposed approach.", "other_comments_or_suggestions": "- It might be interesting to explore more long-context tasks in addition to LongBench. Recently there have been a variety of long-context benchmarks such as RULER and HELMET, which cover a wider range of tasks and types of contexts.\n- Llama 2 is used as the base model, which has a limited context window. It would be interesting to explore whether the proposed approach can help more recent models that have already extended to long contexts (such as Llama 3 and Qwen models).", "questions_for_authors": "- In the proposed method, there are some key hyper-parameters that need to be decided, such as the group size and local window size. I'm wondering how they are decided and what are their effects on the model's performance? It would be nice to include more analyses and ablation studies.\n- I'm also wondering how much data will be needed to fine-tune the model with the proposed approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5rN6gmwwQ", "reviewer": "Reviewer_XMmi", "summary": "This paper addresses the challenge of long-context language modeling by proposing a Core Context Aware Attention (CCA-Attention) mechanism. The method improves efficiency by dynamically selecting core tokens within token groups while preserving local information. CCA-Attention consists of two main components: (1) a Globality-aware Pooling Module, which segments the input into groups and selects representative core tokens to approximate global attention, and (2) a Locality-preserving Module, which maintains a local attention window to retain fine-grained token interactions. Experimental results demonstrate improvements in both latency and memory usage.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The paper only compares against training-free methods, important training-based approaches are missing. While the method achieves better latency and memory efficiency, the actual improvement in average score is marginal.\n\nA1. We appreciate your emphasis on contextualizing our work against training-based methods. Below, we address the concerns raised:\n\n**Method Positioning and Performance** \nOur work focuses on **long-context modeling via efficient attention mechanism** for pretrained LLMs. This is distinct from position interpolation methods or long-context adaptation techniques like LongLLaMA, which do not modify attention in LLMs. By reducing computational costs ( **7.9×** on 128K context) while outperforming training-free baselines, our method offers immediate value to practitioners. Last, we emphasize our goal is to reduce redundancy in attention, which should be compared with those that also optimize attention (e.g., MInference).\n\n**Further Experiments**\nTo address your concerns, we further compare our method with a training-based model, LongLoRA[ICLR 2024] with PI techniques. In Table I, **our CCA-LLM outperforms LongLoRA in modeling accuracy, inference speed and memory footprint**. We will include these in the revised manuscript.\n\nTable I: Results on LongBench-E. ''FTL'' denotes the latency to generate the first token.\n|Model|Avg.↑|FTL↓ (s)|Mem.↓ (GB)|\n|-|-|-|-|\n|LLaMA2-7B-32K|22.11|9.15|35.58 |\n|• LongLoRA|21.58|9.15 (1.0x)|35.58 (0%↓)|\n|• CCA-LLM (Ours)|**21.86**|**2.59** (**3.5**x)|**19.12** (**46**%↓)|\n\n** Note: During inference, LongLoRA exhibits the same computational and storage overhead as full self-attention. The reason is LongLoRA's S2-Attention is only applicable in training and uses full self-attention during inference.\n\n\n> Q2. More results on different architectures.\n\nA2. According to your suggestions, we conduct more experiments on LLaMA3.1-8B and Qwen2.5-7B (128K context). The results in Tables II and III show CCA-Attention outperforms state-of-the-art counterpart MInference in terms of **performance, computational efficiency**, and **memory reduction** (see Table II and III).\n\nWe would include these results in the revised paper.\n\nTable II. Comparisons of Llama3.1-8B-Instruct-128K on LongBench-E.\n|Model|Avg.↑ (%)|FTL↓ (s)|Mem.↓ (GB)|\n|-|-|-|-|\n|Llama3.1-128K|37.93|9.55|40.38|\n|• MInference|37.74|4.93 (1.9x)|35.95(11%↓)|\n|• **CCA-LLM (Ours)**|**37.81**|**3.08** (**3.1**x)|**20.63** (**49**%↓)|\n\nTable III. Comparisons of Qwen2.5-7B-128K on LongBench-E.\n|Model|Avg.↑ (%)|FTL↓ (s)|Mem.↓ (GB)|\n|-|-|-|-|\n|Qwen2.5-128K|38.38|10.58|35.11|\n|• MInference|36.72|4.86 (2.2x)|32.40 (8%↓)|\n|• **CCA-LLM (Ours)**|**38.08**|**2.74** (**3.9x**)|**19.31** (**45**%↓)|\n\n> Q3. Benefits of using fixed-size grouping. \n\nA3. This allows for optimized memory access and parallel computation. While dynamic group sizes would introduce significant implementation complexities.\n\n> Q4. If multiple tokens in a group have high scores with the last token, wouldn't using one core token to represent them introduce significant approximation errors?\n\nA4. The mentioned issue can be significantly mitigated by the fine-tuning process, based on the finding: CCA attention with/without fine-tuning yields performance of 6.92 and 22.24 on LongBench-E, respectively. This confirms core token aggregation effectively captures group-wise information.\n\n\n> Q5. Why are the scores in Table 1 different from the official leaderboard?\n\nA5. The leaderboard tests **chat models** with SFT on LongBench, while we test **base models** without SFT on LongBench-E. \n\n> Q6. Whether during the fine-tuning only fixed g and s are used? How can it perform well across varying g and s?\n\nA6. We use fixed $g$ and $s$ during finetuning. The reasons for the promising performance with varying $g$ and $s$ are twofold:\n- Globality-aware pooling module dynamically **pools groups of tokens into core tokens based on their importance, regardless of $g$**. Thus it generalizes to arbitrary $g$ during inference.\n - Locality-preserving module employs rotary position embeddings that encode relative distances rather than absolute positions, which **decouples from specific $s$**, allowing adaptation to different $s$.\n\nWe would include these in the revised paper.\n\n> Q7. Was CCA-LLM fully or partially fine-tuned in Tables 1 & 2? Could you compare them?\n\nA7. Full fine-tuning. In Section C.3, we compare different fine-tuning strategies. Full fine-tuning achieves better performance than partial fine-tuning (22.24 vs. 21.96) on LongBench-E with more training hours (15.4 vs. 11.2).\n\n> Q8. Implementation details in Figure 4.\n\nA8. We use batch size=1, $g$=16 and $s$=1024.\n\n\n> Q9. Why does MInference barely reduce KV cache usage?\n\nA9. Minference's sparse attention is only applicable in pre-filling. During decoding, it uses full self-attention, leading to same KV cache usage.\n\n---\nWe sincerely hope our clarifications above have addressed your questions.", "claims_and_evidence": "While efficiency gains are demonstrated, the actual performance improvement in terms of model accuracy/quality is \"marginal\" compared to baselines. Without seeing the exact numbers, it appears the efficiency-performance tradeoff may not be as compelling as suggested.\nGeneralizability claims: The experiments are limited to two LLaMA-7B models, making it difficult to conclude that these benefits would extend to other architectures or larger models.\nMethodological effectiveness: Several technical aspects lack sufficient justification:\nThe rationale for fixed-size grouping in the pooling module\nThe potential information loss when using a single core token to represent groups with multiple high-attention tokens\nThe ability to perform well across varying parameter settings when trained on fixed values\nBenchmark performance: The reviewer highlights significant discrepancies between reported results and official leaderboard scores on LongBench tasks, calling into question the validity of the performance evaluation.\nTechnical clarity: Several implementation details are insufficiently explained, including training strategies, parameter configurations, and variable definitions.", "methods_and_evaluation_criteria": "The methods and general evaluation approach make sense for the problem of efficient long-context modeling, but several gaps in the evaluation methodology and presentation limit the strength of conclusions that can be drawn from the results. The paper would benefit from more comprehensive model testing, clearer parameter specifications, and better explanation of observed performance patterns.", "theoretical_claims": "The paper takes a primarily experimental approach to validating its method, with theoretical understanding coming from empirical results rather than formal proofs.", "experimental_designs_or_analyses": "The experimental design appears to have several validity concerns, particularly regarding:\n- Transparency of experimental parameters\n- Benchmark result discrepancies\n- Limited model testing\n- Unclear training methodologies\n- Potentially biased baseline comparisons\n\nThese issues substantially impact the ability to verify the paper's claims.", "relation_to_broader_scientific_literature": "The paper make a relevant contribution to the efficient long-context modeling literature, though with the methodological and evaluation limitations.", "essential_references_not_discussed": "The paper only compares against training-free methods, implying that important training-based approaches are missing. For example:\nPosition Interpolation methods (PI, NTK-aware PI)\nLongLLaMA and other specifically fine-tuned long-context models\nThe absence of these references, particularly other competitive training-based methods, creates a significant gap in contextualizing the paper's contributions. By only comparing against a subset of training-free methods (and not the strongest ones), the paper fails to properly position its approach within the current state of the art for long-context modeling, potentially overstating its relative advantages.", "other_strengths_and_weaknesses": "Strengths\n1. The paper focuses on an important and timely research problem—efficient long-context modeling.\n2. Combining the long-range Globality-aware pooling module with the short-range Locality-preserving module intuitively makes sense and is quite innovative.\n3. The paper is also well-written, and the illustrations are clear and detailed, making it easy to follow.\n﻿\nWeaknesses\n1. The baselines used for comparison are all training-free methods, yet the average performance improvement is relatively small. While the proposed method achieves better latency and memory efficiency, the actual improvement in average score is marginal. A discussion on why this is the case would be helpful.\n2. Experiments are only conducted on two LLaMA-7B models. To better demonstrate the generalization of the proposed method, it would be beneficial to evaluate it on larger models or different architectures.\n3. What are the benefits of using fixed-size grouping in the Globality-aware pooling module? If multiple tokens within a group have high attention scores with the last token, wouldn't using one core token to represent them introduce significant approximation errors?\n4. The scores in Table 1 seem unreasonable for tasks such as single document QA, multi-document QA, and summarization because there is a significant discrepancy compared to the official leaderboard in https://github.com/THUDM/LongBench/tree/main/LongBench. Could the authors clarify the reason for this difference?\n5. In Figure 3, CCA-LLM uses different group size g and local window size s. The authors state that the group size g ranges from 16 to 2, and the local window size s ranges from 4096 to 1024. However, the specific parameter values used for each point are not annotated in the figure, which causes confusion given the apparent significant differences in performance. Additionally, I'm curious whether during the fine-tuning only fixed g and s are used; how can it perform well across varying g and s settings?\n7. CCA-LLM’s training strategy in Table 1 and Table 2 is unclear. Was full fine-tuning or partial fine-tuning used? It would be helpful if the authors could provide results for different fine-tuning strategies to allow readers to better understand the trade-off between training cost and model performance.", "other_comments_or_suggestions": "1. Some variable definitions are unclear. For example, in line 225, what does s represent? Although Algorithm 1 mentions it as the local window size, it should be explicitly defined the first time it appears in the main text. Similarly, how is K_1^G obtained? Does it refer to the first dimension of K^G? A clearer explanation is needed.", "questions_for_authors": "1. What is the batch size in the experiment shown in Figure 4? It appears that CCA-LLM can handle context lengths up to 128K without significantly increasing kv cache usage compared to full self-attention. What are the configurations of s and g here? Moreover, MInference barely reduces kv cache usage, which is also confusing. Could the authors clarify it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces an innovative approach to long-context language modeling, called Core Context Aware (CCA) Attention, which significantly reduces computational complexity and storage overhead while maintaining full token accessibility. The CCA-Attention mechanism dynamically groups tokens and compresses them into core tokens based on their significance, helping to focus on semantically important content and reducing redundancy. This novel approach is designed to be a plug-and-play mechanism for existing LLMs, offering compatibility and flexibility while achieving superior performance in both computational efficiency and memory usage reduction. Experimental results demonstrate the method's effectiveness, with a 7.9× speedup and a 93% reduction in memory footprint over full self-attention on 128K context, without compromising accuracy.
The main concern about the paper is the novelty which has been discussed in the rebuttal, mainly with reviewer JpcZ.  Although I agree that the paper's novelty is not such significant, I think the plug-and-play feature of the method is new and demonstrates its efficiency compared with existing work.  So I would give a "weak accept" to this paper.
Decision: Accept (poster)