Long Context Compression with Activation Beacon
Authors: Peitian Zhang
Zheng Liu
Shitao Xiao
Ninglu Shao
Qiwei Ye
Zhicheng Dou
Venue: ICLR 2025 Poster (2025)
Keywords: Context Compression
Long Context LLMs
LLM Memory
Primary area: foundation or frontier models, including LLMs

Abstract: Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts. To achieve this, our method introduces the following technical designs. 
1) We directly compress the activations (i.e. keys and values at every layer), rather than leveraging soft prompts to relay information (which constitute a major bottleneck to encapsulate the complex information within long contexts).
2) We tailor the compression workflow, where each fine-grained input unit is progressively compressed, enabling high-quality compression and efficient computation during both training and inference. 
3) We train the model through compression-based auto-regression, making full use of plain texts and instructional data to optimize the model's compression performance.
4) During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations. 

Extensive evaluations are conducted on various long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Needle-in-a-Haystack. Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains a comparable performance to the uncompressed baseline across various scenarios, 
achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache.

Published as a conference paper at ICLR 2025
LONG CONTEXT COMPRESSION WITH ACTIVATION BEACON
Peitian Zhang'.2 Zheng Liu't Shitao Xiao Ninglu Shao 1.2 Qiwei Ye Zhicheng Dou?
1: Beijing Academy of Artificial Intelligence,
2: Gaoling School of Artificial Intelligence, Renmin University of China (namespace.pt. zhengliug 02 gma coom
ABSTRACT
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. this paper, we propose Activation Beacon, plug-in module for transformer- based LLMs that targets effective, efficient, and flexible compression of long contexts, To achieve this, our method introduces the following technical designs. 1) We directly compress the activations (i.c. keys and values at every layer), rather than leveraging soft prompts relay information (which constitute major bot- tleneck to encapsulate the complex information within long contexts). 2) We tailor the compression workflow, where each fine-grained input unit is progres- sively compressed, enabling high-quality compression and efficient computation during both training and inference. 3) We train the model through compression- based auto- regression, making full use of plain texts and instructional data to op- timize the model's compression performance. 4) During training, we randomly sample a compression ratio at each step, teaching the model to support wide range of compression configurations. Extensive evaluations are conducted on var- ious long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Keedle-in-a-Haystaee Whilst existing methods struggle to handle these challeng- ing tasks, Activation Beacon maintains a comparable performance to the uncom- pressed baseline across various scenarios, achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache.
INTRODUCTION
Large language models (LLMs) need to process long contexts to accomplish many important tasks, such as long-document understanding (Jiang et al. 2024b), long-content creation (Bai et al. 2024).
and long-term memorizatiion/ressoiing (Zhang et al. 2024). To address these needs, modern LLMs are built with extended context windows (e.g.. 128K) that enable remarkable long-context processing capabilities (OpenAI) [2024] Yang et al., 2024 et al.) 2024). Despite their effectiveness, LLMs encounter efficiency challenges processing long contexts. On one hand, transformer-based LLMs incur substantial computational costs due to the quadratic complexity of self attention. On the other hand, they require tremendous GPU memory to hold the KV cache of the entire sequence for faster decoding. Both computation and memory costs increase as the context length grows.
A wide array of studies are dedicated to alleviating efficiency issues, among which context compres- sion is promising direction (Mu al. 2023 Chevalier al., 2023 Ge et al. 2024 Diang al.) 2023alb) b). This approach aims to compress raw input into more concise representations, allowing the generation process to be conditioned computation cost of inference and memory cost from KV cache, while also enabling the processing of longer inputs than the LLM'' built-in context window.
a shorter context. Therefore, helps reduce both
Despite the current progresses, it remains a tough challenge to compress long contexts. Specifi- cally, existing methods usually summarize the context into few soft tokens (Chevalier al. 2023 Ge et al. 2024), which constitute the major bottleneck to summarize the complex information within
"Peitian Zhang and Zheng Liu are the co-first authors
Zheng Liu is the corresponding author

---

Published as a conference paper at ICLR 2025
there
be
light
and
there
was
light
Let
there
(b)1
be
light
(b)
{b)
(b)
and
there (b)
was
light
(b)
Chunk
Chunk
Figure 1: Overview of Activation Beacon. The context is partitioned into chunks. Each chunk is further split into fine-grained units and interleaved with beacon tokens according to compression ratio (2 in the figure). The LLM encodes one chunk time, compressing the context into beacon tokens' activations, which are accumulated and reused for encoding following chunks.
long contexts. Besides, they try to compress the context "all-at-once", lacking fine- tneeggaaied han- dling of the detailed information. Moreover, these soft tokens must be re-encoded before generation, resulting in inferior efficiency in both training and inference. Lastly, these methods are learned to compress with a fixed number of soft tokens, thus, it's hard to customize the compression ratio for downstream tasks. While some alternamtive methods focus deleting unimportant tokens (Jiagg jet al. 2023b Li et al. 2024b), they depend on the input question to estimate the token importance, limiting their efficiency in real-world multi-turn scenarios.
To address the above challenges, we present Activation Beacon (Figure D plug-in module to transformer-base based LLMs that enables effective, efficient, and flexible compression of long contexts. Activation Beacon is featured with the following technical designs.
First of all, we introduce new special token, called the beacon token (b). The context is distilled into beacon tokens' activations (i.e. keys and values at every layer), whose capacity are large enough encapsulate the complex information within long contexts.
Next, we tailor the compression workflow, where each fine-grained context unit is progressively compressed. Specifically, the long context is partitioned into equal-size chunks. Each chunk is fur- ther split into fine-grained units of size where a is the desired compression ratio. A group of beacon tokens are interleaved with these units (one beacon token is dispatched to the end of every unit). The LLM encodes one chunk at time, distilling the chunk's 's information into beacon tokens' activations during self attention. After encoding, the raw tokens' activations are discarled;; while the beacon tokens' activations are accumulated and reused for encoding following chunks. This pro- gressive workflow brings forth several advantages: 1) It can handle inputs longer than the backbone LLM's context window the chunk size is small. 2) It achieves fine-grained compression since the attention scope each beacon token is diffcrentiated. 3) By caching and reusing activations, it facilitates contiguous gradient propagation in training, avoids re-encoding overhead in inference, and allows for incrementally updating the compression results in multi-turn scenarios.
Finally, Activation Beacon is learned with compression-based rego-regresiion optimize the gen- eration quality conditioned on the compressed context. Thanks to high sample efficiency, the model be effectively trained with 1B plain corpus and 30K fine-tuning samples (maximum context length is 20K), which can be quickly accomplished. During training, we randomly sample the compression ratio for each chunk, enhancing the model's flexibility tackle different compression ratios in downstream tasks. Note that all beacon tokens share the same token embedding, one can use arbitrary number of beacon tokens to achieve the desired compression ratio by repeating.
In our experiments, Activation Beacon is applied to Llama-2 (Touvron et al. 2023) and Qwen- (Yang et al. 2024). We evaluate the resulted models on a variety of long-context tasks (whose lengths may be much longer than the training length, e.g., 128K), such as document understanding, few-shot learning, and Needle-in-a-Hassteel
Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains comparable performance to the uncompressed baseline across various compression configurations, meanwhile achieving 2x acceleration and 8x KV cache reduction. Moreover, the LLM's original capabilities short context is well preserved.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Recently, processing long context has become fundamental capability of modern LLMs (OpenAl 2024 et al. 2024| Yang et al. 2024 AeepSeck-Al 2024). The recipe of context window extension roughly the same: modifying the rotary position embedding (Su et al., 202I) by extrapolation and interpolation (Chen et al. ing long-dependence data in both the pre- pra-tingn and post-traiming stage. Despite the impressive progress in effectiveness, LLMs face significant challenges in efficiency. There is significant compu- tational cost due to the quadratic complexity of transformer, and huge memory cost because LLMSs need to hold the KV activations of the entire sequence on GPU for faster decoding. Multiple threads of research endeavour to reduce these costs, which are discussed as follows.
2023a ntk 2023 Peng et al. 2023 Ding et al. 2024). and leverag-
Sparse A Attention. Conventional sparse attention methods require re-training model from scratch using the designated sparse patterns (Zaheer et al. 2020; Beltagy et al., 2020). However, extensive recent studies have identified that the attention pattern of LLMs are naturally ly sparse despite they are densely trained (Jiang et al.] (2024a) Xiao et al. 2023 Han et al. [023] Zhu et al. (024). They also propose dynamically set appropriate sparse patterns for each head so that the attention mass be largely preserved, leading to competitive performance against the full-attention method with reduced computation. However, these methods require holding all KV activations determine the optimal sparse patterns, making them unsuitable for KV cache reduction. There are some sparse attention methods that directly eviet the middle tokens (Han et al. 2023 Xiao et al., 2023). Despite their high efficiency and ability to generate endless fluent texts, these methods cannot memorize information in the middle contexts, leading to inferior performances on long- context tasks Xiao et al. (2024).
chip to dynamically
KV Compression. This line of research focuses on compressing the KV activations to reduce the attention computation well as the cache size. Since the KV activations are per-layer, per- head, per-token, and per-channel float numbers, they can be reduced from all the five dimensions (including the numerical dimension). For example, CLA (Brandon et al.) 2024) shares the KV cache across multiple layers; GQA (Ainslie et al. 2023) compresses multiple key/value heads into a single one; MLA (DeepSeek-Al (024) compresses the channels into fewer and more compact ones; and KIVI (Zirui Liu et al. (023) quantizes the numerical value in the activations. The sequence-wise compression (also known context compression), where Activation Beacon falls, is introduced in the following paragraph. It is orthogonal to the compression along other dimensions, and the complementary effect of the compression along different dimension could be left for future work. Besides, some recent studies design efficient strategies for offloading and transferring KV cache (Liu et al. 2023 Xiao et al., 2024). They can also be jointly used with KV compression techniques to achieve more efficient long- conteconext generation.
Context Compression. This type of methods aim to compress the raw context into shorter yet more compact representations. Existing studies are usually tailored for compressing short context (less than K). which tend to be sub-optimal for long-context compression. Specifically, Gisting (Mu) et al. 2023) compresses the user instruction into gist activations all at once. As result, it cannot process context longer than the backbone LLM's sss window. CCM (Kim et al.| (0224) extends Gisting to compress conversations in online chatting, yet it cannot be used in general long context tasks such as long document understanding. ICAE (Ge et al. 2024) and AutoCompressor (Chevalier et al. 2023)
alleviate this problem by segmenting the long context into chunks and compressing each chunk, in order to compress contexts longer than the backbone LLM's window. CEPE (Yen et al. 2024) shares similar workflow while introducing standalone encoder compress the context and utilizing the compression results through a cross-attention module. However, these methods compress the context into soft tokens, which are the major bottleneck to encapsulate the complex information in long contexts. Their compression workflow also lacks fine-grained handling of the chunked inputs, resulting in inferior compression quality. Moreover, these methods must perform re-encoding or employ additional cross- attention mechanism to utilize the compressed soft tokens, which introduces extra overhead. Lastly, since the number of soft tokens are pre-defined, it is hard to flexibly assign the compression ratio for downstream tasks. Another branch of methods (Jiang et al. 2023b Li et al.) 2024b) propose to delete unimportant tokens to realize compression. However, they depend on the input question to accurately estimate the token importance, leading to low efficiency in real-world multi-turn scenarios. Compared with existing approaches, Activation Beacon able to achieve more effective, efficient, and flexible compression. Based on context compression techniques, there
3

---

Published as a conference paper at ICLR 2025
some innovated frameworks like LLoCO TTnn et al. 2024). It is built upon compressor and decoder. where the context is compressed offline and offloaded into a retrieval system. The decoder then efficiently responds to the user imputs based retrieved compression results. Both modules are learned with in-domain fine-tuning. Our work aims at improving the compressor itself, and hence is orthogonal these frameworking research.
3 METHODOLOGY
LLMs accomplish arbitrary tasks in the form of next-token prediction. Formally, given the context X = == [zi...... 2n1, the LLM generates the next token based on all preceding tokens and its well- trained parameters: Pr(x=+1 I1,...,, Tn )) Transformer-based LLMs incur heavy computation cost due to the quadratic complexity of self attention; besides, they require tremendous GPU mem- ory to store the KV cache of Ign+1 for faster decoding (Zhang et al. 2023). Both the costs in computation and memory significantly expand when the context length increases.
Activation Beacon employs a new special token, namely beacon token (b), and condenses the raw context X into beacon tokens' activations V (i.e. their keys and values at every layer). The next- token prediction is converted condition on the compressed context instead of the plain one. Given | < [X] both the computation cost and the KV cache size reduced. Additionally, the LLM is enabled to handle context longer than its window size based on the compressed representations. We tailor the compression mechanism and the learning method of Activation Beacon towards achieving effective, efficient, and flexible compression, which will he elaborated in the following.
3.1 COMPRESSION MECHANISM
Overview. We propose to progressively compress each finennr units of long contexs. Specifi- cally, given the input context X whose length may exceed the LLM's context window N, is first partitioned into chunks of the same size (e.g., 1024):
[1 11,,- -fl....,3r
Partition
X [(-1))+tI
[ i =
Next, for each chunk X,, determine compression ratio og (w is evenly divisible by a.). The chunk is further split into fine-grained units of size a. Then a group of k; w/a; beacon to- kens, B; = [(b)i.... (b) are interleaved with these units. In other words, one beacon token is dispatched to the end of every unit:
X; Interleave B X
[1] 111.- Fa, (b)
-w-a (b)2.)-
(2)
The LLM encodes these chunks one by one, compressing the contextual information of each chunk into the corresponding beacon tokens' activations during self attention. After encoding X we discard activations of all the raw tokens X;. while we accumulate the activations of the beacon tokens B₁. When encoding the next chunk 1 the LLM directly conditions on the accumulated beacon activations proxy to the raw context X
This progressive workflow benefits both compression quality and running efficiency. On one hand, it enables thorough distillation of complex information within long contexts and allows for the com- pression of inputs that exceed the LLM's context window. On the other hand, by caching and reusing beacon tokens' activations, it avoids redudant computation and allows for incrementally update of the compression results in multi-turn interactions.
Encoding and Compression. As shown in Figure 2 Activation Beacon reuses all modules of the L.M except imposing a slight modification on self attention. Without loss of generality, for the i-th chunk X the encoding process can be written as:
LLM(
(b) (b) (b)
ii.....
(b) 2 -
Th (b)
(3)
bescon activations accumulated from X
current chunk X!
where the input to the LLM a mix of the activations accumulated from previous chunks and the tokens to be encoded within the current chunk. Let D denote the LLM's hidden size, H E
'The last chunk Xfc/w] may be shorter than us which is omitted for simplicity.
4

---

Published as a conference paper at ICLR 2025
Forward ①
FFN
x3
<
Layer Norm
693
xf
Forward
x2
69/
Self-Attn
Self-Attn
x3
x]
Layer Norm
010
Figure 2: Activation Beacon performs compression during self attention while reusing all other modules of the LLM. Forward @ encode and compress the first chunk. Forward 2: encode and compress the second chunk conditioned on activations of preceding beacon tokens.
[(=+k,)xD XD denote input hidden states to self attention in an arbitrary layer of the LLM. We first slice out the hidden states of raw tokens and beacon tokens:
I (i o (b)}, 1 (i | (b)}; H H["",, H H11').
(4)
Then the hidden states are projected into queries, keys, and values:
Q W
K = HR
V = W y W
Q W
K w H
where W are the LLM's original projection matrices and W are the newly introduced matrices to handle beacon tokens only. Afterwards, the query/key/value states of raw tokens and beacon tokens are scattered back to acquire Q, K, E (R(++))x
Q[1'] Q", Qu1 Q:: K[1'] K", -[1'-K'; VIr V", V Vb.
(6)
Finally, the standard self-attention is computed over the entire input:
Q ( K) ))
A = softmax mask
V A V}.
VD
In the above equations, f denotes matrix concatenation. Kac Voc R. 1xD are the beacon tokens' activations accumulated from previous chunks where m;-1 =ド kj. and mask denotes the causal attention mask. During self attention, all tokens are encoded by their relative positions (Im411..... ... + w 1] for queries and [o..... m, - 1] for keys). The value states V, further processed by other modules (e.g., output projection, MLP, and LayerNorm) before passing the next layer. After self attention, the keys and values of beacon tokens, i.e. K and Vo, have distilled the contextual information of X,. They are incrementally accumulated:
K = (K K').
V = (Voc; V*J.
our default setting, the beacon tokens are interleaved with raw tokens. This leads to differ- entiated attention scope for each beacon token ((b), attends one more interval than (b) contributing to the fine-grained compression of the context. We also explore the setting to dispatch all beacon tokens at the end of the chunk, which results in inferior compression quality (44.6).
Note that unlike ICAE (Ge et al. 2024) and LLMLingua (Jiang et al. 2023b), Activation Beacon unifies generation and compression operations within single forward pass of the LLM. That is to say, the hidden states of the last input token H [R" [1]] is directly used to decode the next token without resorting another decoder model.
Efficiency Analysis. Activation Beacon reduces the KV cache by a times where a is the average compression ratio and hence the memory cost. This is because it only needs store the compressed
5

---

Published as a conference paper at ICLR 2025
(A] Llama-2-78
(8) Owen-2-78
(C) Qwen-2-726
Methed
Methed
Method
Activation Sebion
Seaioon
L.
3.4
3.5
Contect
Contect
Controt
Figure 3:
Comparison of the forward FLOPS of different models using full attention and Activation Beacon (the compression ratio is annotated in the brackets).
activations of the preceding chunks instead of the raw activations. In terms of computation, the sit- uation a bit more complex. Specifically, Activation Beacon significantly reduces the computation in self attention, because each token only needs to interact with local tokens within the chunk and preceding beacon tokens, which are approximately a times shorter than the raw context. However, it also triggers more computation to encode the inserted beacon tokens in other modules (e.g., MLP). Formally, given an LLM with fixed number of layers, attention heads, and hidden size, let denote the input context length, sPet denote the cached context length, the forward FLOPs is:
FLOPs = F (s, sPet) + (O))(s),
(9)
where F is the computation during self attention, and FOth is the computation of other modules. For full-attention models, $ = n, gget 0. For "beaconed" models, the FLOPs is:
FAu ( (a + 1)w a
(i (6-1))) Fnth + (1)) a
FLOP$ ben
(10)
a
Since the implementation of FAA and FOth depends on the actual setting of the LLM (see Ap- pendix [B, we visualize the FLOPs curve of three different LLMs in Figure 3 can be observed that Activation Beacon consistently saves computational costs across different model settings and scales. The extent of saving amplifies as the context length grows, finally achieving more than x4 reduction 256K context. The specific implication latency is studied in $4.3
3.2 LEARNING METHOD
Compression-Basee Auto-Regression. Activation Beacon is learned to optimize the generation quality conditioned on the mixture of the compressed context and the local context. Formally, the compression-baste next-token prediction loss is minimized:
[N/w]
w
min.
Σ
Σ
Pr(xf (b))....,, (b)
O O*)
(11)
④ denotes the parameters of the LLM itself, which are fixed throughout the training process. Ob includes the projection matrices for beacon tokens at each layer W W W.. and the token embedding of beacon token e(b) (we use one shared embedding for all beacon tokens). The training loss can be obtained from all tokens except the ones in the first chunk. Such a property leads to high sample efficiency that maximizes the use of training data. Note that we exclude the beacon tokens from the above loss (setting their labels to -100 because they are solely intended for compression.
No Stop Gradients. Recurrent memory methods (Chevalier al.) 2023) Bulatov et al. (023) stop the gradients back-propagattio at given chunk number to improve the training efficiency. This is because these methods depend on the final-layer outputs of preceding chunks encode the current chunk, which results in deepened computation graph more chunks are involved. In contrast, Activation Beacon only depends on the previous-layer outputs preceding chunks (the encoding of X layer only conditions on the results of X layer I - 1), which is the same any auto-regressive LLMs. Thus, the gradients can naturally flow through all chunks to optimize the compression effect over long contexts.
Chunk Wise Random Compression Ratio.
To teach the model to flexibly support diverse com- pression granularities, the compression ratio o; for the i-th chunk is randomly sampled from

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation LongBench (Bai al.;
2023).
Activation Beacon maintains comparable outperforming other compression methods.
performance to the uncompressed baseline (Full-FT))
"Length" indicates the number of tokens in the input context.
Model
Method
Length
Single-Doe
Multi-Doc
Summ.
Few-Shot
Code
Full
4K
24.7
22.4
24.6
63.2
57.7
Llama-2 Llama-2-7B 2-7B
Full-FT
32K
34.8
27,5
23.2
61.8
57.8
Cumo.
32K
12.9
16.4
16.3
23.8
39.4
ICAE
32K
19.5
19.2
19.5
24.8
27.8
LongLLML.
32K
21.5
18.8
21.7
49.5
53.2
SnapKV
4K
24.2
22.6
16.3
57.7
Ours
32K
34.9
27.5
25.0
61.4
Qwen-2-7B
Full
32K
38.8
37.5
26.7
70.1
60.3
Full-FT
32K
41.0
40.6
26.8
68.5
66.1
LoLgLLML.
32K
24.7
20.3
26.3
55,9
50.1
SnapKV
32K
38.7
37.6
26.2
67.1
60.3
Ours
32K
40.5
40.3
26.8
68.4
66.4
{2, 4,48 8, 16, 32} during training. At inference, one can choose one compression ratio according to the specific efficiency requirement in downstream tasks and stick for all chunks.
4 EXPERIMENTS
Our experiment mainly study Activation Beacon's effectiveness (44.2), efficiency (44.3). and flex- ibility ($4.4) in long context compression. Besides, we explore Activation Beacon's impact on short-context capabilities of the backbone LLM (44.5) and the effect of each technical design ($4.6)
4.1 SETTINGS
Implementation. Activation Beacon is applied to Llama-2-7B (chat and Owen-2-7B (instruct). The chunk size w is 1024 for Llama-2 and 2048 for Qwen-2. FlashAttention-2 (Dao 2023 is used speed up attention computation. For our experiments, we use Huggingface framework (Wolf et al. 2020 and one 8xA800 (80G) machine.
Training. The training consists of two phases. In pre-training, we use 1B tokens sampled from Red- Pajama [Cmputer 2023). The eos token is appended to the end of every document. In fine-tuning, we leverage Alpaalpaca (Chen et al.) 2023b), BookSum (Kryšcinski et al.) 2022). and synthetic data from GPT-3.5 (details in Appendix A). All the training samples are shorter than 20K. The batch size is 8. The learning rate is 5e-5 for pre- pre-training and le-5 for fine-tuning, with linear decay and no warmup. As introduced, the LLM's original parameters frozen throughout the training process.
We compare Activation Beacon with the uncompressed baseline (denoted as Full) and the uncompressed baseline fine-tuned with the same training data (denoted Full-FT). Besides, include the following context compression methods that can tackle long context for comparison, in- cluding AutoCompressors (Chevalier al. (023), ICAE (Ge et al. 2024), LongLLMMinguu (Jiang jet al., 2023b). and SnapKV V (Li et al.) 20244b). The first two methods only support Llama-2. To guarantee fair comparison, fine-tune their official checkpoints using the same training data.
Baselines.
4.2 COMPRESSION EFFECTIVENESS
To verify the compression effectiveness of Activation Beacon, we evaluate on LongBench (Bai et al. 2023), which consists of a variety of long context tasks with 32K maximum length, including quession answering, summarization, few-shot learning, and code completion. Since Llama-2 has context window of 4K, we truncate the context longer than 4K from middle before inputting to it. For compression methods implemented on Ilama-2, we set adaptive compression ratio, translating to x2 compression for 4K-8K contexts, x4 compression for 16- contexts, and x8 compression for 66K322K contexts. For methods implemented on Qwen-2, 2, we apply a uniform compression ratio of x4. The results are reported in Table T We highligh two observations in the following.
We use I lama-2 2 because AutoCompressor and ICAF based it, both of which important baselines.

---

Published as a conference paper at ICLR 2025
Activation
Activation
12
Dapth Porcent
Depoth Percant
33
5.0
728
20182
3714
R012
74455
A35022
Context Length
Context Length
Figure 4: Evaluation on Needle-in- a- Haystack. Activation Beacon can accurately retrieves the nee- dle most of the time, despite the context is far longer than its training data.
Firstly, Activation Beacon achieves superior compression quality over other compression base- lines across all tasks. Concretely, it siginificantly outperforms ICAE and AutoCompressor, which verifies that several soft tokens are not enough encapsulate the rich information within long con- texts. LongLLMLingua also lags far behind Activation Beacon because it need delete too many tokens given high compression ratio (e.g., x4, x8), which may destroy the coherence the con- text and lose important information. Despite SnapKV's 's top performance among baselines, it cannot compress context longer than the backbone LLM's window. This because it estimates the token importance based self attention, which becomes inaccurate once the context exceeds the window size, limiting its practical usage when compressing long contexts.
Secondly, Activation Beacon achieves comparable performance to the fine-tuned uncom- pressed baseline (Full-FT) even though Full-FT takes the entire context without compression. This indicates that Activation Beacon is able compress long contexts without evident informa- tion loss, which validates its high compression quality yielded from the progressive compression workflow. Furthermore, Activation Beacon improves upon Llama-2 by large margin despite their context window is the same, i.c. 4K. The gain is because Llama-2 (Full) directly uses the truncated 4K context, while Activation Beacon compresses the 32K context into 4K compact activations. This implies that Activation Beacon can effectively introduce useful information from Llama-2's unseen context. Therefore, it can be viewed as an efficient approach for context extension.
We further evaluate Activation Beacon on Needle-in in- a- Haystack (NIAH) following the official set- tings (gkamradt (023) to investigate whether will lose fine-grained information. The accuracy is estimated by ChatGPT (ranges from to 10). For both Llama-2 and Qwen-2, we set adaptive compression ratio as introduced above. The results are shown in Figure 4 can be observed that Activation Beacon precisely retrieves the needle most of the time. Note that Activation Beacon conducts query-independent compression, which means it has no prior knowledge of what to com- press and what not. Hence, this remarkable performance again validates mechanism and learning method can preserve the fine-grained contextual information. Moreover, Activation Beacon is only trained on context shorter than 20K, while its compression capability can generalize to far longer contexts (e.g., 128K).
tailored compression
4.3 COMPRESSION EFFICIENCY
We evaluate the efficiency of Activation Beacon based on the Multi-Needle- in-a-naystack task fol- lowing NeedleBench (Li al.) 2024a). Specifically, we fix the context length to 32K for Llama-2 and 128K for Qwen-2, and insert 3 different needles at different positions. The task is organized in multi-turn conversation setting, where the model asked retrieve one specific needle in each turn. The experiment is repeated 20 times for each model with distinct needle positions. In Table 2 report the accuracy and the end-to-end latency of compression generation (measured in seconds).
It can be observed that Activation Beacon enjoys lower latency than other compression base- lines. Notably, it is 1.8x faster than AutoCompressor because does not have to re-encode the soft tokens from previous chunks. It also leads to 9.3x and 3.6x acceleration upon LongLLMMingua and SnapKV given three turns, respectively. This because both baselines are query-dependen while Activation Beacon is not, which eliminates the need re-compute the compression results for
8

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation Multi-Nedle--i--a-aayyye -aysss- where the questions issued one-by-one in
multi-turn conversation setting. All compression methods use x8 compression ratio. Activation Beacon consistently outperforms other compression baselines while enjoying lower latency, espe-
cially when the context lengthens and the turn number increases.
I-Turn
2-Turn
3-Turn
Model
Length
Method
Acc
Latency
Acc
Latency
Acc
Latency
Full-FT
9.75
1.336
9.45
1.532
9.10
1.726
Qwen-2-7B Llama-2-1h
AutoCompr.
1.60
2.135
1.50
2.561
1.50
2,994
32K
ICAE
2.15
1.182
2.15
1.476
2.00
1.805
LongLLML.
2.05
2.813
2.00
5.062
2.00
7.034
SnapKV
1.00
0.859
1.00
1.656
1.00
2.199
Ours
9.75
1.153
9.40
1.356
9.05
1.638
Full-FT
9.75
4.399
9.50
5.254
9.20
6.153
128K
LongLLML.
2.00
10.455
1.55
19.768
1.50
27.751
SnapKV
9.45
3.955
8.95
7,803
8.85
10.659
Ours
9.70
2.445
9.35
2.773
9.10
2.981
AutoCompeessor
LonglLMLingua
Snappy
(A) Contedd Length-1K
(8) Contect Length-4K
(4) Contect Length-32K
Accunny
Compression Rado
Compression Radoo
Compression Ratio
Figure 5: Evaluation on Needle-in-a-Haystack with various compression ratios based on Llama-2. Activation Beacon achieves top compression quality across all compression configurations.
different input questions. Moreover, Activation Beacon demonstrates consistent speed-up over the Full-FT baseline, achieving acceleration at 128K context length. This matches our estimation in Figure 3(b) as Activation Beacon (x8) saves half of the computation. In the meanwhile, since the compression ratio x8, leads to 8x reduction of the KV cache. Lastly, Activation Beacon always attains nearly-lossless generation quality against the uncompressed baseline, which is in line with previous observations.
4.4 COMPRESSION FLEXIBILITY
Activation Beacon is learned to support various compression ratios during training. In Figure 5 we evaluate its compression quality under different compression ratios and context lengths. According to the figure, Activation Beacon maintains top accuracy across all compression ratios, outperforming most compression baselines by a large margin. Though SnapKV performs on par with our method at IK and 4K context length, it fails to compress inputs longer than the L.MM's window size, which may limit its practical usage. To summarize, Activation Beacon is a flexible solution to long context compression with the support of diverse compression ratios and various context lengths. Generally, we recommend to use x8 compression ratio as preserves most information with high efficiency.
4.5 SHORT-CONTEXT CAPABILITIES
Since Activation Beacon inter- leaves beacon tokens with raw to- kens and is primarily trained with long-context tasks, it is intrigu- ing to examine whether the current recipe will impair the chort-context capabilities of the backbone I.I.M
Table 3: Activation Beacon preserves the short-context capa-
bilities of the backbone LLM.
Model
Method
MMLU
ARC-C
BoolQ
GSM8K
Full
47.5
48.5
86.2
9.2
Llama-2-7B
Ours
46.6
48.4
86.5
9.3
Full
70.1
62.7
87.1
76.0
Qwen-2-7B
Ours
69.1
62.7
87.2
76.2

---

Published as a conference paper at ICLR 2025
In Table we compare Activa- tion Beacon with the original LLM
(Full) on popular benchmarks, including MMLU (Hendrycks et al. 2021), ARC-Challenge (Bhak- thavatsalam et al., 2021), BoolQ (Clark et al. 2019), and GSM8K (Cobbe et al. 2021). We can observe that Activation Beacon leads very little performance degradation on short-context tasks. In other words, the short-context capabilities are well preserved. We conjecture that the primary reason is the LLM's original parameters are frozen throughout the training process.
4.6 ABLATION STUDIES
Table 4: The impact of different technical factors.
We study the impact of each technical factor, including the compression of fine-grained con- text units, the sampling strategy of compression ratio, and training stages. The experiments are based on Qwen-2-7B and Single-Doc QA task from LongBench (32K context with x4 com- pression ratio). The results are shown in Ta- b Firstly, instead of splitting the chunk into fine-grained units and interleaving beacon tokens, we append all beacon tokens at the end of the chunk so that their attention scopes are the same. can be observed that such operation results in significant information loss after compression, which justifies the effectiveness of our fine-grained compression mechanism. Secondly, we replace the chunk- wise random compression ratio with the instance- wise one, which randomly selects one compression ratio for each training instance rather than each chunk. We can observe that the chunk- wise setting facilitates better learning of the compression functionality. Lastly, we remove either pre-training or fine-tuning. It can be observe that both stages are useful, and the combination of both leads to the optimal performance. This also implies that the compression quality of Activation Beacon can be further enhanced given more abundant and targeted training.
Method
Single-Doc
Default
40.5
w/o Fine-Grained Compression
35.2
w/o Chunk Wise Random Ratio
37.7
Pre-training w/o Fine-tuning
34.9
35.5
5 CONCLUSION
This paper introduces Activation Beacon, plug-in for transformer-basee LLMs to enable effective, efficient, and flexible compression of long contexts. Activation Beacon featured for several critical innovations, including the progressive and fine-grained compression workflow to distill the context into a small set of activations, the compression- based auto-regression to optimize the model with high sample efficiency, and the random sampling of compression ratios to support various down- stream scenarios. According to extensive experimental evaluations, Activation Beacon consistently outperforms existing context compression methods across various compression configurations. even maintains comparable performance to the uncompressed baseline, meanwhile achieving 2x ac- celeration and 8x KV cache reduction. Moreover, the short-context capabilities of the LLM is well preserved.
6 ACKNOWLEDGEMENTS
This work is supported by the National Science and Technology Major Project (20232D0121504), National Natural Science Foundation of China No. 62272467, and Beijing Municipal Science and Technology Project No. Z231 1000100003330
REFERENCES
Ntk-aware scaled rope, 2023. URL
coments//112755/ttkaaa scapereesscaeeeeeeeeeeeeeeeeeeee rolow allows 11ama www.ms.deelloooooooosoesss
have/
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6- 10, 2023, pp. 4895 4901. Association for Computational Linguistics, 2023. doi:
10

---

Published as a conference paper at ICLR 2025
10.1 22.18633/1/2023.EMNVV EWNLL-MMINNN9--0 M9I.. /ttp:ttp:://ob.or/00.8885///////
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context llms. CoRR, abs/2408.0705 2024. doi: 10.4550/ARXXV.2448.0005 URL htttp:::/di..ozg///
18550/arXiv 07055
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020. URI. /tt.h/ppss///arrt...r///////////
05150
Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richard- son, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge.
CoRR, abs/2102.03315, 2021. htt.:tt:::/ /ttp::://axxvvvvo wwtp:///g////bbpaoo/ooooo/s /tt.ttttp.e//aaxx..rrr/aa//////p
03315
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan- Kelley. Reducing transformer key-value cache size with cross-layer attention. CoRR, doi: 10-48550ARXXI.2405.1222 (8550/arXiv.24000 2409.12281
abs/2405. 12981, 2024.
URL /ttth::::///// //doi, /otttpp://do...o//1....
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer Im tokens and beyond with RMT. CoRR, 122222344..100,, 2023. doi: 10445550ARXXVV2204..000 URL https: 11062
48550/
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. /2307, 2023b.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 3829-3846. Association for Computational Linguisties, 2023. URL org 2023. ewnlp-mainn 232
https: http://ccanathohooooooo
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio ), Proceedings of the 2019 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 2924 2936.293..222 Association for Computational Linguisties, 2019. doi: N..18653/V1/NS1 1300. URL httphhttpp//ooii.. .og / 111111666vll11
1300
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL
htt.ps://rrxi..orrr/bb///////
www.oorggaaaoo.. /21 14168
Together Computer. Redpajama: An open source recipe reproduce llama training dataset, 2023.
URL https://githui. http:///hhbb..oomooogttteeeeooop
RedP -amaaa
Tri Dao. Flashattention-22 Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691, 2023. doi: 10.48550/ARXIV.2307.000
URL
4 48550/arXiv.
DeepSeek-AL. Deepseek-v2. v2: A strong, economical, and efficient mixture--f-eexperts language model, 2024.
11

---

Published as a conference paper at ICLR 2025
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond million tokens, 2024.
Abhimanyu Dubey et al. The llama herd of models, 2024, URL https://axxiv.org///
2407.211 2177331783
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2024.
gkamradt, 2023.
URL
https://github.. com/ http://wwww....oggggaatattt////a
Needle LnAHaystack
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. CoRR, abs/2308. 16137, 2023. doi: 10. 48550/ARXIV.2308.. 1611177177.0
httt:s/:: //doi www.oorgoo.... 48550 a arxiv 211../.......1133.......1
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URLLtttps://gppnzeeeaa
net www./eeeoeeidniieeeeee
jmI 3GmQ
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: what's 's the real context size of your long-context language models? CoRR, abs/2404.06654, 2024. doi: 10.485550ARXXI.24444.000
URL https:
rtddi.org/1... 48550 2404 06654)
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023a.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yer Lin, Yuqing Yang, and Lili Qiu. LonglImlingua: Accelerating and enhancing llms long context scenarios via prompt com- pression, 2023b.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li. Chin- Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context 1lms via dynamic sparse attention, 2024a. URL
https://arxiv. org/ abs/ 2407 02490
Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmentee generation with contect llms. CoRR, abs//a00. 2024b. doi: 10. 0048550/ARXXV.2200.5555 URL
2406. 15319
Jang Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction, 2024.
Wojciech Kryšciñski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization, 2022.
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in million context window?, 2024a. URL htttppp::////aw.ot/////bbb/////b
19663
Yuhong Li, Yingbing Huang. Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation, 2024b.
12

---

Published as a conference paper at ICLR 2025
Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng. Yuyang Huang, Shan Lu, Michael Maire, Henry Hoffmaan, Ari Holtzman, Ganesh Ananthanarayanan, and Junchen Jiang. Cachegen: Fast context loading for language model applications. CoRR. abs/2310.07240, 2023. doi: 10.44550/ARXI12330000744 URI.
2310.
072400
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. CoRR, 0e/p22304004467,, 2023, doi: 1048550/AARXXY2234000444.7 URL URUattps://ool.oz 10. 00,48550/arxi11 2304. 08467
OpenAl. Gpt-4 technical report, 2024. URL UKL/ittppe//arxxv..or//a///////p
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint (rxxi:2309.0007/ 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding. CoRR, abs/2104.0986/ 2021. URL BtLlttss////xxi..oom
abs/2104. 09864
Sijun Tan, Xiuyu Li, Shishir G. Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gon- zalez, and Raluca A. Popa. Lloco: Learning long contexts offline. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-
16, 2024, pp. 17605-17621, Association for Computational Linguistics, 2024. URL https:
//aclanthology: org /ttp////222222 2024. emnlp-main. 975
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface transformers: State-of- the-art natural language processing, 2020. URL
.rg. awwaaaas111
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free congcconeeett extraplatiin for llms with an efficient context memory, 2024.
wwt.//////bbbssaaa 24611
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint 23Xiv:230..14555 2023.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.106 10671, 2024.
13

---

Published as a conference paper at ICLR 2025
doi: 10.48500AARXIV.240..0000 /ttttpps////...r///0...55/////// 10671
Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 2588-26100 Association for Computational Linguistics, 2024. doi: 10. 18653/v1/200 2olooooo....
URL httptttts//::oooo////
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297 2020.
Zeyu Zhang. Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey the memory mechanism of large language model based agents. CoRR, abs/2404. 13501, 2024. doi: 10.48550/AXXVV244044 13501. URL httL:ttss:///oo /org
005.44550/aaaa0 2404. 13501
Zhenyu Zhang. Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.). Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http::://aapps.aepp nips wttp:pp..cppppppppeppp http://ppppaaelle/pppeaaaaeeaeee httpp///pps/pppee///////eeeeee html
Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, Dahua Lin, and Chao Yang. Sampleattention: Near- lossless acceleration of long context I.I inference with adaptive structured sparse attention. CoRR, abs/2406. 15486, 2024, doi: 10.48550/ARXIV.24666 10. 00044855/arxii 2406 15486
URL URL
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi slug-and-play 2bit kv cache quantization with streaming asymmetric quantization. 2023. doi: 10. 13140/RG. 2.2. 222.22166.37788
URL https://rgdol net/10.
A TRAINING DATA
In the pre-training phase, IB tokens from RedPajama. We add token to the end of each document. The documents longer than 20480 or shorter than 2400 are removed.
Prompt A.1
(Segment 1}
(Segment N}
Question: {Question for Segment 1]
Answer: [Answer for Segment 1}
Question: {Question& for Segment N}
Answer: (Answer for Segment N}
In the fine-tuning phase, we use three datasets. 1) LongAlpaca (Chen et al. 2023b) , which contains long-context QA and summarization data; 2) BookSum (Kryšcinski et al. 2022) which contains chapter-level summarization of books. 3) Synthesized QA dataset. This dataset contains 16K long- context question answering instances (13K for books and 3K for papers). Specifically, split
14

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of the breakdown latency of prefilling and decoding, well peak GPU memory between the full-attention bascline and Activation Beacon, Activation Beacon uses x8 compression uniformly.
Model
Input Length
Method
Prefilling
Decoding
Total
GPU Memory
Full
0.522
0.709
1.231
27.0
32K
Ours
0.514
0.506
1.020
18.1
Qwen-2.5-7B
Full
3.312
2.081
5.393
64.3
128K
Ours
2.038
0.550
2.588
24.6
Full
0.952
1,862
2,814
45.7
32K
Ours
0.996
0.857
1.853
36.0
Qwen-2.5-14B
Full
OOM 40.3
128K
Ours
4.340
1.117
5,457
given long context (a paper or book) into short segments (a chunk with less than 4096 tokens) using For each segment, we prompt GPT-3.5-turbo to generate 4 question- answer pairs based on segment. We then group continuous segments using Template|A.] where we can control the resulted context length by concatenating different number of segments. The books are randomly sampled from Books3 corpus, and the papers are sampled from Arxiv, both coming from the Pile (Gao et al. 2020). All the above fine-tuning data formatted in the manner of multi-turn conversations and limit the context length up to 20480. To mitigate forgetting, we also include 5000 samples from the pre-training data.
the
B FLOPS CALCULATION
Denote the input sequence length 8, the cached sequence length sPet, query head number as h4, key/value head number as h²,, the hidden size D, head dimension as intermediate size 7, and vocabulary size V.
AMt pako
+ Fat Esoftman + Fav + pout
Egke x s x D x d 13 +2 +2x2xxxx2 Xx
Fos 2 x h x 5 sPd)) x d
psoftman
119 (s pat)
x
Fou h x saa)
pout
(12)
5oth FuD + pgate + Fdown + plm
FND 22 x 2x1
pgate =XXXI
pdown = -XXXX DxI
Fim =2xsx
(13)
C ADDITIONAL EFFICIENCY ANALYSIS
We further evaluate the efficiency of Activation Beacon by decomposing the latency of pre-filling and decoding. Specifically, we set the context length to 32K and 1 128K and enforce the model to
15

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of the training time and the peak GPU memory usage during training between
Full-FT and Activation Beacon.
Model
Method
DeepSpeed Stage
Training Time
GPU Memory
Full-FT
Zero-2
6.32
51.2
Qwen-2.5-7B
Ours
Zero-2
6.58
38.5
Full-FT 7ero-3 (OOM w/ 7ero-2)
18.67
79.4
Qwen-2.5-148
Ours
Zero-2
12,34
75.6
Table 7: Evaluation on RULER (Hsieh et al., 2024). Initially, Activation Beacon lags behind full- attention models on saasoning/aggrggation tasks, yet the gap can be easily compensated by addi- tional fine-tuning with synthetic 200 samples.
Model
Method
NIAH AVG
VT
CWE
FWE
QA AVG
Full
79.06 80.13
88.00
41.04
66.67
40.25
Full-FT
71.95
32.28
64.76
52.38
Qwen-2.5-7B
Ours
78.43
25.30
10.12
60.00
52.15
Ours + Synthetic FT
80.91
85,30
59.30
72.18
51.27
generate precisely 128 tokens. We uniform x8 compression ratio. The peak GPU memory during the entire generation process is also reported.
According Table [ method accelerates both pre-filling and decoding, and the acceleration extent amplifies as the context gets longer. Mcanwhile, our method is better at speeding up decoding because directly conditions on the beacon tokens' activations, which are 8x shorter than the raw activations used by the baseline. Moreover, Activation Beacon significantly reduces the peak GPU memory usage, enabling efficient processing of long context on customer GPUs.
Lastly, we also compare the training efficiency of Activation Beacon against the standard fine- fine-tuning. The results are shown in Table The experiment result indicates that Activation Beacon achieves comparable training speed as the Full-FT baseline while significantly reducing the memory cost.
D RULER EVALUATION
In addition to LongBench and NIAH, we evaluate the performance of Activation Beacon on RULER (Hsieh et al., 2024), challenging long-context benchmark that consists of 13 synthetic tasks. Concretely, there are 6 tasks extending the regular NIAH to multi-key, multi- and multi- query variants: task (Variable Tracking) aiming at examining the multi-hop reasoning capability by tracking variable assignments; 2 tasks (Common/Frequent Words Extraction) targeting the ag- gregation capability by counting the word occurrences; and 2 tasks performing question answering while inserting noisy context the ground-truth passages. Our evaluation is based on 128K context length and x4 compress ratio for Activation Beacon. The results are shown in Table 7
be observed that our initial Activation Beacon model (denoted "Ours") maintains a com- petitive performance on QA tasks, while lagging far behind the full-attention baseline on reason- ing/aggregation tasks (VT and CWE). A similar drop can also be observed on Full-FT, too. One likely reason for this disadvantage is that our current fine-tuning recipe only uses one-hop QA data stated in Appedix A)). which does not teach the model to perform complex reasoning or count- ing, resulting in inferoo performance on these tasks. However, this problem should be mitigated by adjusting the composition of training data. We add 200 synthetic samples (100 for VT and 100 for CWE) with 20K maximum context length to the training data and fine-tune the model. The new model, denoted as "Ours+Synthetic FT", achieves substantial improvements on both VT and CWE while preserving its competitive performances other tasks. This observation further validates the effectiveness of our Activation Beacon: can quickly learn the desired compression capability given a limited set of training data.
16

---

Published as a conference paper at ICLR 2025
Table 8: Evaluation of 7B and 14B models LongBench (Bai et al. 2023
Activation Beacon
always maintains a comparable performance to the expensive full-attention fine-tuned bascline.
Model
Method
Length
Single-Doc
Multi-Doc
Summ.
Few-Shot
Code
Full
32K
41.9
45.2
26.5
69.1
64.9
Qwen-2.5--B
Full-FT
32K
42.7
46.1
26.7
67.6
66.3
Ours
32K
42.5
45.8
26.8
67.4
66.4
Full
32K
42.5
52.9
71.7
66.7
Qwen-2.55 2.5-14B
Full-FT
32K
43,9
50.5
27.1
68.8
67.1
Ours
32K
43.4
49.9
27.1
68.5
67.4
E EXPERIMENTS ON LARGER MODELS
We apply Activation Beacon Qwen-2.5-7B and Qwen-2.5-14B inspect its performance larger models. The LongBench evaluation results (with uniform x4 compression ratio) are shown Table can be observed that Activation Beacon retains its effectiveness, holding comparable performance to the full-attention fine-tuned baseline.
Additionally, we report the training & inference efficiency of 14B models in Table and Table 6 respectively. Activation Beacon significantly reduces the memory costs during training, enabling efficient training of large models at high speed. Besides, stably accelerates the inference process while being much more memory-efficient than the full-attention baseline.
17


Reviews:
[{"review_id": "5xNgFZh6Re", "reviewer": "Reviewer_ft7Q", "summary": "The paper introduces “Activation Beacon,” a plug-in module to conduct long-context compression for LLMs. The proposed approach progressively compresses the activations at each layer and can be trained in the conventional auto-regressive way of language modeling. The authors demonstrate the benefits of this approach through evaluations on various long-context tasks for compression quality and inference efficiency.", "questions": "- How are rotary embeddings managed for the beacon tokens? Although the LLM processes a fixed chunk at a time, the relative positions of the beacon tokens vary across chunks. How are positional embeddings applied in these cases?\n- Additional parameters are added and fine-tuned for self-attention projections specific to the beacon tokens. What is the impact of these added parameters on VRAM usage and latency? If the cost is significant, could LoRA fine-tuning be effective for the proposed activation beacons approach?\n- What portion of time is allocated to prefilling and decoding? While the proposed method reduces some recomputation, it may require customized attention masks or iterative context processing, which could lack efficient kernel implementation or result in extra kernel calls. Please provide a latency breakdown of prefilling and decoding for specific workloads (e.g., 32/128k context, 128 decoded tokens) and compare it with the flash attention full-context baseline.\n- How does the proposed approach affect fine-tuning throughput? Please compare its performance with Full-FT.\n\nI am open to adjusting my ratings if all concerns and questions are adequately addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Compressing by chunks at each layer avoids the need for recomputation and addresses gradient back-propagation challenges present in some prior baselines that rely on recursive dependencies from final-layer outputs. This design enhances both training and inference efficiency.\n- The chunking approach and the interleaved insertion of beacon tokens are straightforward and intuitive.\n- Evaluations on various benchmarks indicate that the proposed approach generally outperforms the KV cache compression and “soft-prompt” compression baselines, achieving notable reductions in both inference time and memory usage.\n- Training with randomly sampled compression ratios enables flexible compression ratios during testing.", "weaknesses": "- In addition to LongBench and NIAH, it is essential to evaluate the proposed approach on newer, more challenging benchmarks, such as RULER [1].\n- Some recent context compression baselines, including CEPE [2] and LLoCO [3], are not discussed in the paper and should be included for a more comprehensive discussion or comparison.\n\n[1] Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? COLM 2024.  \n[2] Yen et al. Long-Context Language Modeling with Parallel Context Encoding. ACL 2024.  \n[3] Tan et al. LLoCO: Learning Long Contexts Offline. EMNLP 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6OaZ9DtPj", "reviewer": "Reviewer_JN1J", "summary": "The paper introduces \"Activation Beacon\", a compression method designed to enhance long-context processing efficiency in LLMs. The approach compresses the activations of keys and values in transformer layers, avoiding bottlenecks associated with traditional soft prompt methods. Additionally, a progressive compression workflow compresses each context unit in chunks, allowing the model to handle longer contexts than the original LLM's window. Experimental results show Activation Beacon achieves significant memory and computation savings, with minimal loss in performance.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Activation Beacon reduces inference time by 2x and KV cache memory costs by 8x compared to the uncompressed baseline.\n\n2. The method supports adaptive compression ratios, allowing flexibility for different tasks and contexts.\n\n3. The proposed model maintains short-context capabilities, preserving the performance of the original LLM.", "weaknesses": "1. The performance of this method may vary with model size. Current evaluations focus on medium-sized models, lacking validation on larger-scale models, leaving its effectiveness and applicability in very large models underexplored.\n\n2. The added complexity of managing beacon tokens and compression ratios increases implementation overhead for end-users, particularly when adapting to different tasks. In addition to actual inference latency, specific memory usage data across implementations would help clarify practical resource requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBt3dp5vQn", "reviewer": "Reviewer_9Nqc", "summary": "This paper compresses activations (keys and values) rather than using soft prompts, facilitating a progressive, fine-grained compression process. Specifically, it first partition input into small chunks, interleaving special beacon tokens that accumulate contextual activations.", "questions": "overall, this paper is novel and idea is well presented. please add more techniques for comparison so that users can choose different method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an efficient method to compress long contexts, reducing memory usage by up to 8x and speeding up inference by 2x.\n- Its progressive, fine-grained compression approach maintains high compression quality, allowing the model to handle longer inputs than its built-in context window.\n-It supports flexible compression ratios, preserving model performance across various long-context tasks without degrading short-context capabilities.", "weaknesses": "- Lack of Comparison with KIVI: The paper does not provide a direct comparison with KIVI, a relevant compression method that could offer insights into the performance trade-offs.\n- GPU Time Omission: The paper does not report GPU training or inference time, leaving uncertainty around the practical computational cost and efficiency of the proposed method.\n- Scalability Concerns: The method requires 8 A800 GPUs to train a 7B parameter model, raising concerns about its scalability to larger models like 70B, where computational demands could become prohibitive.\n- Limited Comparative Analysis: The paper would benefit from including more baseline methods, particularly compression-based approaches like KIVI, KV-layer shared compression methods such as CacheGen, and relative-position encoding strategies like LM-Infinite.\nAdditional References Needed: Incorporating comparisons with relevant works, such as LM-Infinite [1] for dynamic context management, CacheGen [2] for efficient context loading, and KIVI [3] for asymmetric quantization of KV caches, would strengthen the evaluation and highlight the advantages and limitations of the proposed approach.\n\n[1] LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models\n[2] CacheGen: Fast Context Loading for Language Model Applications\n[3] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRyUnSKkY6", "reviewer": "Reviewer_4UYt", "summary": "The paper introduces a method called Activation Beacon for efficient long-context processing. The method adds learned \"beacon\" tokens at regular intervals in the input query. These tokens are expected to learn \"summaries\" of the text. At inference time, when processing long contexts, the beacon tokens are retained and the other context tokens are discarded. Thus, the beacon tokens essentially provide a summary of the context. The authors evaluate their method in comparison with a few other recent methods for efficient long context processing. Their method significantly improves results on LongBench and Multi-Needle-in-a-Haystack. The authors also provide ablations for various design choices.", "questions": "My main question is in the \"regarding differences with previous works\" above. I want to understand if the results are improved mainly from decreasing chunk size, or if there's another difference between soft tokens and beacon tokens that explains the difference.\n\nAlso, what window size do you use? From Table 1, your model has a context length of 32k. I'm guessing you use this window size, but I don't see it explicitly stated, and line 184 suggests that 1024 would be a common window size, so I'm not sure. Since LongBench has only a few examples above 32k, I'm guessing the window logic isn't really used much (unlike for Needle In a Haystack)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on an impactful area (long-context efficiency for LLMs).\n- The paper provides a relatively simple idea that is well-explained. I view simplicity as a plus - if a simple idea can give strong accuracy improvements, it's far better than an unnecessarily complicated idea.\n- The paper demonstrates strong results. Table 2 demonstrates strong accuracy at good latency on standard benchmarks for long context. Their method is competitive with full fine-tuning and better than baselines. Table 1 provides strong accuracy as well (though latency is missing).\n- The figures do a good job of explaining what's going on. Figure 1 and Figure 2 give nice overviews of the method.\n- The method is computationally efficient compared to fine-tuning. Their \"pretraining\" (starting from an already-pretrained model) only requires 1B tokens which is very few.\n- The paper ablates design choices (Table 4).\n- The paper is generally well written.", "weaknesses": "- In Table 1, it's not obvious whether the latencies are comparable. The compression ratio isn't mentioned.\n- line 368: why do you use adaptive compression for llama-2 and uniform compression for qwen?\n\nMy main perceived weaknesses are regarding differences with previous works, and understanding why this method is performing so well:\n- line 135: \"ICAE and AutoCompressor... segment the long context into chunks and compress each chunk. However, both of them compress the context into soft tokens\" <- how are these soft tokens different than beacon tokens? (similarly, on line 373-374, you mention soft tokens being a drawback)\n- line 137: \"Their compression workflow also lacks fine-grained handling of the chunked inputs, resulting in inferior compression quality\" <- it seems like all they would need to do to allow \"fine-grained handling of the chunked inputs\" is just choose a smaller chunk size, so that the soft tokens appear more frequently. Is that right?\n- - If this is true, it seems like your main contribution is the insight that soft tokens should be distributed evenly through the context. Would doing this massively improve the accuracy of ICAE and AutoCompressor? It seems like this is the main discovery, but I'm left wondering if I'm missing some more fundamental difference.\n\n[Minor]:\nline 47: \"it it\" -> \"it\"\nline 53: \"alternamtive\" -> alternative\nline 371: \"highligh\" -> \"highlight\"\nline 483: \"scope\" -> score\nTable 2: give units of \"latency\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)