SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator
Authors: Guoxuan Chen
Han Shi
Jiawei Li
Yihang Gao
Xiaozhe Ren
Yimeng Chen
Xin Jiang
Zhenguo Li
Weiyang Liu
Chao Huang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Language Modeling
Attention Mechanisms
LLM Architecture
Sparse Attention
KV Cache Compression
Primary area: deep_learning->large_language_models
TL;DR: A new language modeling perspective by utilizing punctuation marks

Abstract: Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator
Guoxuan Chen 12 Han Shi
Jiawei Li
Yihang Gao
Xiaozhe Ren Chao Huang?
Yimeng Chen
Xin Jiang'
Zhenguo Li
Weiyang Liu
Abstract
2.75
275
Vinills
Vanilia
Large Language Models (LLMs) have exhibited exceptional performance across spectrum of nat- ural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and infer- ence speed, due to their quadratic complexity. In this work, we have identified a key pattern: cer- tain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to at- tention scores compared to semantically meaning- ful tokens. This observation suggests that infor- mation of the segments between these separator tokens can be effectively condensed into the sepa- rator tokens themselves without significant infor- mation loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accel- erates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training ac- celeration. Experimental results across training- free, framsinggfromm-seracch, and post-training set- tings demonstrate SepLLM's effectiveness. No- tably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining com- parable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up million tokens or more while maintain- ing consistent language modeling capabilities.
2.05
2.50
S
2.0
2:65
Sip
2.90
2.400
1,36x
2.55
2.45
2.
2.40
150000 200000 Training Time
160000
TFLOP%
Figure 1. The loss comparison between vanilla Transformer and the proposed L. ScpI.I achieves lower loss w.r.r different computation costs and different training time consistently.
tasks, including natural language processing (Zhang et al., 2020; Raffel et al., 2020), computer vision (Dosovitskiy et al., 2020), and scientific machine learning (Geneva & Zabaras, 2022). However, vanilla Transformers that rely on next-token prediction face significant computational chal- lenges, particularly when scaling to larger models and longer contexts. These computational inefficiencies significantly impact both inference speed and training time.
The core challenge underlying these efficiency issues is the self-attention module, which exhibits quadratic complex- ity with respect to the number of input tokens. Research efficient Transformers in LLMs primarily follows two major directions. The first approach focuses on linear atten- tion (Katharopoulos et al., 2020; Schlag et al., 2021), replac- ing the vanilla self-attention module with alternatives that achieve linear complexity. However, these modifications make the architecture significantly different from traditional self-attention, preventing direct utilization of powerful pre- trained Transformer models. The second approach empha- sizes KV cache optimization (Xiao et al., 2024a; Zhu et al., 2024; Xiao et al., 2024b; Li et al., 2024), aiming to climinate redundant KV cache to accommodate longer input contexts. For example, Xiao et al. (2024a) introduced an adaptive mechanism that selectively retains essential tokens and their KV based on cumulative attention scores. Similarly, Zhu et al. (2024) proposed token selection strategy with con- trolled sparsity, achieving near-lossless acceleration. While promising, these training- ffee methods adapt poorly to the training stage, resulting in discrepancies between training and inference performance. StreamingLLMM (Xiao et al., 2024b) represents a notable attempt to address these lim-
1. Introduction
Transformer-basee models (Vaswani et al., 2017) have ex- hibited exceptional performance across a wide range of
'Huawei Noah's Ark Lab "The University of Hong Kong 'Center of Excellence for Generative AI, KAUST "Max Planck Institute for Intelligent Systems, Tübingen. Correspondence to: Han Shi <hi han huawei.com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Accelerate LLMs by Compressing One Segment into One Separator
new language modeling perspective and simple yet effective framework accelerate inference.
Through targeted masking experiments on well-trained LLMs, we demonstrate that separator tokens contain cru- cial information and are essential for model performance. This finding suggests that sequences are initially seg- mented by separators, with segment information being compressed into these prequently-attendae separator to- kens while redundant specific tokens can be discarded.
The visualization for attention scores of different layers given the input "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. ...". Note that the separator tokens like" " and "" contribute massive attentions.
Figure 2.
We conduct comprehensive experiments to validate Se- PLLM's effecttivnness across various tasks, datasets, and backbone models, examining performance in training- free,
and post-training settings.
itations by preserving attention sinks and local tokens to reduce computation and memory overhead. However, it omits many intermediate tokens, resulting in performance degradation compared to standard Transformers.
We have made our implementation publicly available at repllm.githuh.ce Our codehase supports efficient multi- node distributed training with accelerated attention mod- ule Sep-Artention and also supports numerous existing Fusion Operators to accelerate the training process, such as fused rope (Su et al., 2024), fused layer norm, etc.
To gain better understanding of the intrinsic mechanism of LLMs, we analyze attention patterns across different sam- ples. Figure 2 illustrates the attention distribution when Llama-3-8B-instruet (Dubey et al., 2024) processes a math problem (complete results are presented in Appendix A). Surprisingly, rather than focusing on semantically meaning- ful tokens (such as nouns and verbs), L.LMs tend to prioritize attention seemingly "meaningless" separator tokens (like "." or "A "\n")) for information retrieval. This observation sug- gests that segment information is compressed and embedded into these separator tokens, enabling efficient information retrieval without direct extraction from content tokens.
2. Related Work
KV Cache Compression. Recent research has focused on overcoming LLMs' limitations in processing extensive con- textual inputs. FastGen (Ge al., 2024) proposes adap- tive KV cache management method, optimizing memory usage by customizing retention strategies for different atten- tion heads. SnapK V (Li et al., 2024) enhances efficiency through KV cache compression, utilizing attention scores to select and cluster significant positions. H₂O (Zhang et al., 2023) implements a dynamic token retention policy, balanc- ing recent and historically important information optimize memory use. StreamingLLM (Xiao et al., 2024b) expands LLMs' capabilities to handle infinite sequence lengths with- out fine-tuning, by reserving attention sinks and local to- kens. QuickLLaMA (Li et al., 2025) proposes to evict the query-aware KV cache for inference acceleration. Pyra- midInfer (Yang et al., 2024) and PyramidKV (Zhang et al., 2024) modify the KV cache capacity across different layers, prioritizing larger allocations in the lower layers while re- ducing those in the upper layers. However, most works in this category cannot be applied into training phase.
Inspired by this observation, we introduce SepLLM, a new language modeling perspective as well as an efficient trans- former architecture featuring data-dependent sparse atten- tion mechanism that selectively retains only initial, neigh- boring, and separator tokens while dropping other tokens. The training-freee SepLLM performs comparably to vanilla Transformer, which validates our hypothesis that segment information is effectively compressed into separator tokens. More importantly, we integrate SepLLM into the training stage (including both training from scratch finetuning) and implement hardware-efficient kernel based on Flex- Attention (PyTorch, 2024). This integration reduces the dis- crepancies between training and inference that are present in previous approaches. As demonstrated in Figure 1, Se- pLLM consistently achieves lower loss compared to vanilla Transformer given the same computational costs or training time. Moreover, SepLLM reduces computational costs by 28% and training time by 26% while achieving the same training loss. Our contributions are summarized as follows:
Sparse Attention. attention matrices by limiting attention to predefined pat- terns, such as local windows or fixed-stride block patterns. Beltagy et al. (2020) combine dilated local window atten- tion with task- global attention. BigBird (Zaheer et al., 2020) proposes a complxxty attention alterna- tive using global tokens, local sliding-window attention, and random attention. In comparison, SparseBERT (Shi et al., 2021) proposes a differentiable attention mask algorithm to learn the attention mask in an end-to-end manner. Note that most works about sparse attention are using fixed masks and
Sparse attention involves creating sparse
We analyze attention patterns by visualizing token-level at- tention scores, revealing that initial, neighboring, and sep- arator tokens consistently receive high attention weights. Such empirical findings motivate us to propose SepI.L.M,
2

---

Accelerate LLMs by Compressing One Segment into One Separator
built BERT (Devlin et al., 2019) families. In comparison, our proposed S is mainly built on GPT (Brown et al., 2020) series and its attention masks are data-dependent.
A
B
C
D
3. Method
E
gooonite
E
3.1. Fundamental Design
P
A
E
F
G
From Figure 2, we can observe that within a given input context, seemingly "meaningless" separator tokens receive higher attention scores compared to tokens with actual se- mantic meanings. Therefore, we propose a novel Trans- former architecture where, for certain layer of the Trans- former (i.e.,a self-attention layer), each token in the input can only see a portion (not all) of the hidden states of to- kens preceding the current token, outputted by the previous transformer layer. This subset of tokens includes a number of initial words (e.g., attention sinks (Xiao et al., 2024b)), all the separator tokens before the current token, and the closest tokens to the current token. Details are as follows.
In
gevembe
A
X
33
F
Ya
A B
Initial Tokaas
Separatern Tokens
Neighboring Tokens
Figure 3. The overall paradigm of SepLLM. The left side illustrates the attention mask in the training pre- pre-filling stage given the input "ABC, nn". The right side illustrates the KV cache management the generation stage.
former with full attention. See more in Appendices H and I.
Neighboring Tokens. Language tasks usually exhibit strong local dependencies and interactions, since adjacent tokens often form coherent phrases or have dependencies that are required to be captured. Neighboring tokens usually help form locally smooth and coherent contexts, allowing the model to generate sentences that are reasonable within the immediate context. The neighboring tokens, also referred to as local attention or sliding-window ered in various efficient Transformers (Xiao et al., 2024b; Zhang et al., 2023) and we have also adopted this approach, with the number of preceding tokens closest the current token denoted as "n".
Initial Tokens. When using the sliding window mechanism (Beltagy et al., 2020) for generation, removing the key- value (KV) pairs corresponding to the initial tokens in the KV cache results in a noticeable increase in the perplexity of generated tokens, a phenomenon mentioned by Xiao al. (2024b). The initial few tokens are also referred as attention sinks. We retain this setup and further validate the role of initial tokens in subsequent experiments. Usually, a initial tokens are kept.
window attention, are consid-
From Figure 2. we can observe that within a given input context, seemingly "meaningless" sep- arator tokens (such as commas, periods, exclamation marks, semicolons, etc.) that segment sequences receive higher at- tention scores compared to semantically meaningful tokens (such as nouns or verbs). Therefore, we hypothesize that these separators may compress the information of the text segments naturally segmented by them, such that when the Transformer generates new tokens, it only needs to refer- ence the information contained in these separators extract the information pertaining to those text segments. Hence, in training-fres frree scenario, we employed this strategy and achieved similar results to the original model based on full attention across many tasks. Furthermore, to reinforce the ef- fect of using separators to compress information within their respective segments, we employed training-froo from-scratch and post-training approaches to compel the model during training to restrict the current token from accessing all in- formation from distant preceding text, i.e., in each segment, only the separator representing its segment is visible to the current token (with other tokens being masked, see Fig- ure 3). After training in this manner, the information within segments is forced to be condensed into the separators, lead- ing the Transformer's probability distribution for predicting the next word closely resembling that of the original Trans-
Separator Tokens.
3.2. Overall Pipeline
We split the overall pipeline of our proposed SepLLM into training/pre-fillin stage and generating stage. We also provide a theoretical analysis about the Universal Approxi- mation of SepLLM in Appendices and K.
Training/Pre-illingg During the training/pre-fllligg stage of SepLLM architecture, we do not need to multiply all query vectors corresponding to tokens in the input context with all the key vectors. It is sufficient to just multiply the vectors of the query-key pairs corresponding to the high- lighted elements in the mask matrix shown in Figure 3. The formulation can be illustrated in the following.
A =oStmax (A), A= Mul (Q, K T
M)
ddg
0 A V
where Q E RRmxdd K E mx are the matrices of query and key for one attention layer, in which each row vec- tor Q K2 correspond to the query of i-th token and the key of j-th token in the input context with sequence length m. d2 denotes the dimension for key and query vectors.
3

---

Accelerate LLMs by Compressing One Segment into One Separator
Initial Cache
Separator Cache
Past Window Cache
Local Window Cache Sizchorst w
Runtime Cache Usage
S120mi S a edi
Sizeepp S P4
Streport.w
Sizerun eg.. 22
Iteration
A
B
In
F
G
H
dmmp
drop
Iteration 2
A
B
E
\n
F
G
H
10
Compress
Once: Suze
Iteration
B
drrsy
\n
F
G
H
I
\n
Cmop
Iteration 4
B
\n
F
G
H
\n
NNeihhooing Tolorss a = Sizquurr Statheesi
Capacity: og_12
Text
A
B
Figure Overall framework of proposed SepLLM tailored for streaming applications. The KV pairs are storaged four cache blocks (displayed four columns), and are updated in each iteration (shown in single row). Once the runtime usage Sizerun reach the max capacity e, SepLLM move KV caches separator tokens in Past Window Cache into Separator Cache and drop other KV caches.
A,A Rmxm are the raw and final attention maps, re- spectively. V E Rmxd is value matrix of dimension d and O E Rmndd denotes the output for the current atten- tion layer. Mul(-) represents sparse matrix multiplication function which can be optimized by methods like Zhu et al. (2024) and we also implement our own module named Sep- Attention to accelerate this process. M E |m* is a binary mask matrix used as a parameter for Mul(-):
applications. discussed in Fundamental design (Sec- tion 3.1), SepLLM can save substantial amount of KV cache by retaining only the KV for separator, neighboring, and initial tokens. However, the number of input tokens increases, the number of separators in KV cache will also accumulate endlessly, which not feasible for streaming settings. Therefore, we propose Tailored Streaming Design for streaming scenarios.
Aij- Q K
KKKVVaa
de if M.j = 00 if M.j =
Framework. Figure 4 illustrates the SepLLM's process- ing architecture for streaming applications. The diagram depicts multiple iterations, with each row representing a distinct processing step. The system simultaneously main- tains four specialized cache blocks: Initial Cache, Separa- tor Cache, Past Window Cache, and Local Window Cache. Specifically, Initial Cache captures the attention sinks pro- posed by Xiao et al. (2024b). Local Window and Past Win- dow Caches store the KV for consecutive tokens, with Past Window Cache serving as an overflow buffer for the Lo- cal Window Cache. Separator Cache retains the KV for separators which contain condensed segment information.
(2)
where Ar,, Ar.j, A., are the elements in the i-th row and j-th column of matrices A, A, M, respectively. Since Ai,j if Ai.j = -0, the tokens that are not Initial, Sep- arator, and Neighboring tokens will be masked by A V in Equation 1. This strategy (Equation 1) applies all heads of multi-head attention (Vaswani et al., 2017).
The management of the KV cache during the generation stage for this Fundamental Design (Section 3.1) also intuitive. As shown in the right side Figure 3, when generating new token, we only preserve the KV cache for the Initial, Separator, and Neighboring tokens. Therefore, the KV cache in the proposed SepLLM is much smaller and requires less memory. Ideally, based on SepLLMM, the perplexity of generating the next word is comparable to that of the original Transformer with full attention.
Generation.
To describe the cache management strategies, denote the runtime usage of the four caches Sizemit Size Sizepust. and Size respectively. The runtime usage across all KV caches is defined as Sizenm := Sizeinit + Size + Sizepast + Sizecocal which satisfies Size I c. The number of continuous Neighboring tokens defined as n Sizcpuu. + Size Notably,
function of the input sequence length m (together with the specific input dataset D) rather than a fixed hyperparameter for streaming setting. For clarity, we detail the preset hyperparameters of this caching system as follows (Note: + s + < c).
3.3. Tailored Streaming Design
In real-world scenarios, there are numerous streaming appli- cations such as multi-round dialogues, where long interac- tions are expected (Xiao et al., 2024b). Hence, we expect SepL.LM to handle infinite input without significantly sacri- ficing efficiency and performance, especially for streaming
c: The maximum capacity of the entire KV cache.
a: The maximum capacity of Initial Cache.
'B = (0. 1}. which a binary set.
s: The maximum capacity of Separator Cache.

---

Accelerate LLMs by Compressing One Segment into One Separator
The maximum capacity of Local Window Cache. Notably, is also the minimum value of after runtime KV cache usage Sizerm reaches c for the first time.
all other configurations are the same as the corresponding settings Pythia (Biderman et al., 2023). Specifically, the training epoch is set to 1.5 epoch (143000 steps with the global batch size as 1024), which means about 300B tokens in total are utilized for training from scratch, which is identical to Pythia (Biderman et al., 2023).
During streaming sequence generation, SepL.LM will firstly fill Initial Cache and then Local Window Cache. After Sizeloal w reaches w, subsequent tokens are directed to Past Window Cache. Compression is triggered when Sizenn reaches (iteration I in Figure 4), where separator tokens in Past Window Cache are moved to Separator Cache and other tokens are discarded. When Separator Cache reaches its capacity at some total input length mo. enters periodic pattern. Specifically, for m > Ma, n follows a periodically linear function bounded by w and c - a s. The detailed evolution of KV caches is illustrated in Appendix B. To analyze the average usage of KV cache, we define A.. (kt). According to the linearity and periodicity, we have
Parameter Setting. The official 93,000 93.000-step checkpoint of Pythia- Sihii--.4B-dedupee 4eddddd model is used to conduct post- training, which corresponds to just completing one epoch of training on the deduped Pile dataset (Gao al., 2020). And [*.,, " ... "?", "!", tokens used for all evaluations. More specific experimental settings are introduced in the respective experiment sections.
", "(t", "\n"] are separator
4.2. Training-Free
m
We evaluate the proposed SepLLM architecture in the training-free tasks based on the popular Llama-3-8B- Instruct model (Dubey et al., 2024).
lim Rm 77 12-900 DC
(3)
2
For the average runtime KV cache usage of infinite-length sequence generation, we have
Benchmarks. The representative and commonly-used GSM8K-CoT (Cobbe et al., 2021) and MMLU (Hendrycks et al., 2021)) are adopted. DSSM8K-Col (Cobbe et al., 2021) tests model's ability to solve mathematical problems by mooblem-solving skills. CoT (Wei et al., 2022) means the ability to simulate a reasoning process by breaking down complex problems into a series of logical steps. And the default 8 shots are adopted. MMLU (Hendrycks et al., 2021) assesses a model's general knowledge and reasoning ability across a wide range of sub- jects, such history, science, mathematics and so on. The commonly- 5-shot setting is used for MMLU.
lim Sizena- = lim Am
+
+
m=000
m m00
w + <.
(4)
evaluating its reasoning and step-by-stey
2
Positional Encoding. Our positional encoding strategy for streaming settings is the same as the tate-of-the-art StreamingLLM (Xiao et al., 2024b), designed specifically for infinite-length inputs, where we focus on positions within the cache instead of those in the original text.
4. Experiments and Results
Results. The experimental results for training free are shown in Table 1. "Vanilla" represents the original Llama- 3 model with full attention, while "StmmLL' represents StreamingL (Xiao et al., 2024b). a means the number of KV for Neighboring Tokens we retain. For SepLLM, all the KV for Separator Tokens are kept and for the setting SepLIMM (n=256), we find that SepLLMM exhibits compa- rable performance in both multi-step mathematical CoT task and multidisciplinary knowledge reasoning tasks, when compared to the full-attention Llama-3. SepLLM achieves this using only 47.36% of the KV utilized by the original Llama-3 for reasoning. indicating SepLLM's capability of modeling both the contexts requiring multi-step logical anal- ysis and those involving multi-domain knowledge reasoning while retaining only 50% original KV.
4.1. Experimental Settings
We evaluate our proposed SepLLM on the following tasks, ie., training-free, training-frood fonge-froo--tttccl post-training, and streaming applications.
Model. Two popular model families, i.e., Pythia (Bider- man et al., 2023) and Llama-3 (Dubey et al., 2024), are em- ployed for evaluation. Specifically, Pythia- Eythia-160m--eduu 160mddddueed is used as the backbone in the training- teeee-foom--rrre since the model, data, configurations, and checkpoints are all open-source and the training results are reproducible. As for post-training settings, we take Pythia- 1.ahia-1.44-- 1.4B- as our backbone model. Even though Llama-3 exhibits power- ful performance on various downstream tasks, the training experimental details are not available. Therefore, we only use Llama-3 for training-free and streaming tasks.
tasks
L (n=256) setting corresponds to removing all sepa- rators' KV from SepLLM (n=256) setting, except for those in Neighboring and Initial tokens. We observe a notice- able decrease in both mathematical analysis and multidisci- plinary knowledge reasoning abilities for StrmILMM (n=256).
Training Datasets. In the training-frome from--crrachh and post- training tasks, the deduplicated Pile (Gao et al., 2020) is utilized for training, which contains about 207B tokens. And
5

---

Accelerate LLMs by Compressing One Segment into One Separator
GSMSK-CoT
MMLU
flexible
strict
r.KV (%)
humanities
stem
social
other
Overall
r.KV (%)
Vanilla
77.79
77.26
100.00
60.49
56.61
76.50
72.19
65.72
100.00
StrmLLM (4=380)
70.89
71.42
47.54
57.73
54.46
74.39
70.13
63.39
52,50
StrmLLM (x=256)
69.67
68.61
26.00
62.10
54.49
73.06
69.78
62.10
37.73
SepLLM (m=256)
77.18
77.18
47.36
57.66
56.49
76.21
72.19
64.68
44.61
Table 1. Evaluation results and average runtime KV cache usage for training- farenngg--eeeeeeeo experiments on GSM8K CoT 8-shots and MMLU
5-shots. For ScpLLM and StreamingLLM, three initial tokens' KV are kept for this experiment. r. (%) here represents the ratio of KV
usage at runtime for the respective method compared to Vanilla. See more results in Appendices and Table 17.
Method
ARC-c
ARC-e
LBD--pl
LBD-acc
LogiQA
PIQA
SeiQ
Atten. (%)
r.KV (%)
Vanilla
20.14
46.80
34.83
33.28
23.81
62.84
81.50
100.00
100.00
Strml.LMMm=644)
20.65
47.39
44.03
26.74
21.97
63.82
75.80
16.58
15.28
SepLLM(x=64)
19.62
46.46
40.08
28.97
26.42
63.82
80.10
25.83
25.40
ScpLLM(7=1188)
19.97
47.35
30.16
33.18
22.73
64.64
82.60
35.64
32.27
SepLLM(#=64.11)
20.73
48.44
36.54
30.45
25.35
64.36
80.60
32.01
31.58
Scp.LLMnn=64HH/TT
21.42
47.26
33.41
32.80
22.73
63.98
81.20
38.18
37.75
Table 2. The performance of downstream tasks and the average runtime KV cache usage in the traiiig-frrom-seee setting.
Visiila stemi (embo))
et al., 2023) (see Section 4.1). And following Pythia (Bider- man et al., 2023), we conduct tests on the following down- stream tasks: ARC-Challenge and ARC-Easy (Clark et al., 2018), LAMBADA (Paperno et al., 2016) (for Perplexity and Accuracy), LogiQA (Liu et al., 2021), PIQA (Bisk et al., 2020), SciQA (Welbl al., 2017). From the loss curves depicted in Figure 5 and the downstream performance in Table 2, we draw the following analysis.
tembi)
31
SesLL.
S
Septi immb4, 11/1)
M (m1121)
Vantii
Lawe
SepLEM (e-60)
27
SeplEM (e-128)
SpiLLM
1113 TRL0G,
Aliandox
10
(a) Loss w.r.f steps
(b) Loss Ratio w.r.t FLOPPs
Figure 5. Training loss curves training from scratch. 5(b) shows the ratios of the loss values of different methods to that of Vanilla with respect to FLOPs.
Neighboring Token Benefits. Based on the experiments with the settings SepLLM (n=64) and SepLLM (n=128), we find that during training, increasing Neighboring Tokens (n) leads to faster decrease in the training loss curve (Figure 5). Furthermore, models trained with larger exhibit stronger performance in downstream tasks (Table 2). This highlights the important role of neighboring tokens contextual lan- guage modeling and downstream task inference.
L (n=256) utilizes only 26.00% and 37.73% of the KV for the GSM8K and MMLU tasks, respectively, which are less than SepLLM (n=256) (47.36% and 44.61% re- spectively). Consequently, we increase the n of StrmLLM to 380, aligning the kept KV GSM8K to be equal to Se- pLLM (N=256) (approximately 47%, while on MMLU task, S (n=380) retains 52.5% of the KV, significantly higher than SepLLM (n=256)). This leads to improved performance compared to StmmLLM (n=256). However, it still remains lower than the full-attention Llama-3 and Se- PLLM (n=256). This indicates that the KV of separators indeed encapsulates information contained within their re- spective segments, and removing them significantly impacts the Transformer's understanding and reasoning abilities.
Hybrid Layer Benefits. We find that employing certain hybrid architecture is beneficial to both the training loss and the performance on downstream tasks. For instance, by mod- ifying only the first self-attention layer to full attention in the experiment corresponding to SepLLM (n=64) (denoted as SepLLM (n=64,H)), there is moderate optimization in both the training process and downstream tasks. If both the first and last attention layers are changed to full atten- tion (denoted as SepLLM (m=64, this optimization becomes more pronounced. For example, LAMBADA per- plexity decreases from 40.08 for SepLIMM (n=64) to 36.54 for SepLLM (n=64,H) and 33.41 for SepLLM (n=64,H/T),
4.3. Training from Scratch
We train the original Pythia- Pythia-160m-dess 160m-mdpmmd model well as the Pythia-160m-dees 160m-deduped model modified with the Se- PLLM (and StreamingLLMM architecture on the Pile dataset for 143,000 steps using a global batch size of 1024 (involv- ing approximately 300B tokens in total for training). All training configurations are consistent with Pythia (Biderman
The experiment with the setting Strm- LLM (n=64) corresponds to SepLLM (n=64), but does not consider separators other than those in Neighboring and Ini-
Separators' Role.
6

---

Accelerate LLMs by Compressing One Segment into One Separator
Arch.
StrmLLM
SepLLM
Vanilla full
3.0
SepL.LM (1-64, lager
Setting
n=
n=64
n=128
n=64,H
R/T
Sept L.M (x-128)
FLOPs (%)
70.11
71.77
72.58
72.83
73.90
100.0
2.8
SepL.M (1-64)
Vauilla before post-
Atten. (%)
6.43
17.21
22.48
24.11
31.01
100.0
2.6
Loss
Table 3. The comparison of FLOPs and Attention Map Ratios.
2.4
tial tokens. We observe a significant slowdown in the train- ing loss decrease for StrmLLM (n=64), and its performance deteriorates across various downstream tasks. This indicates that the KV corresponding to separators indeed contain in- formation about the segments they belong to, which are beneficial to predicting subsequent tokens.
2.2
2,9
93000
100000
105000
110000 115000 120000 125000
#Iteration
Figure 6. Training loss curves for the post-training setting.
PG19
1M
1.5M
2M
2.5M
3M
3.5M
4M
We also investigate the FLOPs and Attention Map Ratio (indicating the proportion of 'I's in the lower triangle of the attention mask) required by the different architectures when trained on the same input data. As shown in Ta- ble 3, We find that SepLLM can significantly reduce FLOPs by approximately 30%. After plotting the loss ratios be- tween SepLIMM and Vanilla under the same FLOPs (see Figure 5(b)), we observe that SepLLM has lower loss than Vanilla. This indicates that our SepLLM architecture at least has a comparable ability extract useful information from the dataset during training Vanilla, Besides, the detailed wall-clock time per iteration and the wall-clock time speedups are illustrated in Appendix C and Figure 1.
StrmLLM
39.5
38.2
38.3
37.6
36.4
35.8
36.1
ScpLLM (s=32)
37.7
36,6
36.6
36.0
34.9
34.2
34.5
SepLILM (x=64)
37.1
36.0
36.1
35.4
34.3
33.7
33.9
Table 4. The perplexity comparison on the PG19 test set (Rae et al., 2020). For fair evaluation, we keep the entire KV cache capacity e as 324 and Initial Cache capacity a as for both StreamingL.LMM and SepLLM. w=224, s=32/6 for SepLLM.
The results are shown in Table 4. We can observe that for the same KV cache capacity c, the average perplexity of predicting the next token through SepLLM remains consis- tently lower than that of streamingLLMM (Xiao et al., 2024b) within the range from 1M to 4M input length. This once again verifies the ability of KV corresponding to separa- tors to compress segment information and their impact on predicting the probability distribution of the next token.
4.4. Post- test-Training
Since training from scratch is time-consuming, we also conduct post- post-training experiments using 93000-step Pythia- 1 1.4B-deduped checkpoint officially released by Pythia (see Section 4.1 for details). Figure 6 displays the loss curves for post-training, where SepLLM (n=64, larger Ir) denotes we employ an entire cosine learning rate scheduler (including warm-up process starting from 0) identical to that of original 1.biia-.----eduueed from step 0. SepLLM (n=64) and Se- PL..M (n= /28) utilize cosine learning rate scheduler that continues to decay from the 93000th step. From Figure 6, it is evident that increasing n and appropriately raising the learning rate both facilitate the decrease in loss. Moreover, this also illustrates that SepLLM can achieve swift trans- formation from full-attention LLM checkpoint to model that aligns with the requirements of the ScpLLM architec- ture's 's embedding distribution through post-training.
We also test the end-to-end inference time of Vanilla, StreamingLLM and our SepLLM on PG19 (Rae et al., 2020) test set based on LlaMA-3-8B (Dubey et al., 2024). Based on the aforementioned settings, we used these L.LMs to generate 20K and 64K tokens to evaluate their total infer- ence time (wall-clock time), average perplexity, and av- erage runtime KV cache usage. For both SepLLM and StreamingL_MM, the maximum whole KV cache capacity was set to 800 (i.e., c=800), and the Initial Cache capacity was set to 4 (i.e., a=4). For SepLLM, we additionally set s=64 and w=256. The results are shown in Table 5.
Those results demonstrate that our SepLLM can achieve lower perplexity with less wall-clock time well as lower average runtime KV usage, especially for longer sequences, given the same max KV cache capacity c.
4.5. Streaming Applications
4.6. Ablation Study
SepLLM can also adapt well to streaming applications, where infinite-length interactions may occur. Here, we fol- low StreamingLLM (Xiao et al., 2024b) to validate the scenarios of Infitite-enngth interactions using our Tailored Streaming Design on the commonly used PG19 dataset (Rae et al., 2020), which comprises 100 extensive literary works.
Hy Hyperparameters. We conduct various ablation experi- ments specifically for long-input applications. This includes a detailed study of the impact of various hyperparameters across different text lengths (5K to 20K). The experimental results about s and (w,c) pair are illustrated in Table 6 and Table respectively. The conclusions are as follows.
7

---

Accelerate LLMs by Compressing One Segment into One Separator
Length
Methods
r.Kv
ppl
time (s)
Method
w
r.Kv
5K
10K
15K
20K
Vanilla
20K
10K
302.6
523.8
320
324
324
13.18
11.51
8.85
8.91
20K
StrmLLM
800
800
31.5
341.2
StrLLMM
512
516
516
12.87
11.37
8.74
8.78
SepLLM
800
562
28.3
325.8
796
800
800
11.96
11.01
8.67
8.72
Vanilla
64K
32K
1090.8
3380.6
224
324
308
13.01
11.17
8.67
8.72
64K
StrmLLM
800
800
37.9
1096.0
SepLLM
320
516
452
12.91
11.26
8.67
8.72
SepLLM
800
562
33.4
1049.7
512
800
690
12.09
11.03
8.56
8.62
Table 5. The average perplexity and running time comparison on the PG19 test set (Rae et al., 2020). r.KV means the average runtime KV cache usage in the generation process.
Table 7. Average downstream performance (ppl) over different input lengths and average runtime KV usage with different c.w on WikiText, in which a=4 for both methods and s=64 for SepLI.M.
5K
10K
15K
20K
r:KV
Method
initial shift
5K
10K
15K
20K
r.KV
32
13.11
11.31
8.74
8.79
292
StrLLLM
13.2
11.5
8.9
8.9
324
48
13.03
11.26
8.70
8.76
300
StmLLMM
X
/
14.6
13.2
10.8
10.9
324
64
13.01
11.17
8.67
8.72
308
StrmLLM
/
x
425.5
513.1
509.5
506.8
324
Table 6. The perplexity and average runtime KV cache usage of L. with respect to different Separator Cache capacities (s) on WikiText (Merity et al., 2017). in which a=4, w=224, c=324.
StrmLLM
X
x
409.4
540.5
527.5
558.2
324
SepLLM
/
V
13.1
11.3
8.7
8.8
292
SepLLM
X
V
14.9
14.3
12.4
12.5
290
SepLLM
/
x
192.7
214.6
175.0
174.4
292
s: From Table 6, the capacity of Separator Cache affects the perplexity of long-text inference, as we find that in- creasing s leads to certain degree of perplexity reduction.
SepLLM
x
x
226.4
264.7
227.5
228.8
290
Table 8. The perplexity and average runtime KV cache usage of SepLLM and StreamingLLM tested on WikiText (Merity et al., 2017). c=324, a=0/4 for both methods. s=32,=2224 for ScpL.LM
e and w: As can be seen in Table 7, and w can impact the average perplexity in the scenario of long streaming input with lengthy text. Moreover, as they increase, the perplexity decreases accordingly.
of hyperparameter. Given that the types of common separa- tors are limited (e.g., the ones we adopt here [".", ".. "?", "m u." 6." "I "tt", "(n"), we recommend including all commonly used ones. A detailed discussion of the ablation studies on separators is provided in Appendix G.
Settings. We also perform the following experiments to validate the effectiveness of Initial Tokens and Positional Encoding's 's shifting for both StreamingLLM and SepLLM. The experimental results are shown in Table 8 and the dis- cussions are as follows.
4.7. Comparison with More Baselines and Variants
Naive Baseline. We compare SepLLM with more base- lines and variants to verify its effectiveness. The results shown in Table 9 are based on naive baseline in which H, out of H attention heads are configured as sliding window attention, while the remaining heads use full attention. We use MMLU as the benchmark to measure the differences in reasoning ability, ensuring that the runtime KV usage for all methods is kept consistent as possible. It be observed that this naive head- wise baseline performs poorly (regard- less of how many full attention heads are retained), which is due to the heterogeneity among the heads. In contrast, SepLLM achieves performance very close to Vanilla across various knowledge domains with the same KV usage.
Initial Tokens are crucial for modeling context of long streaming inputs, whether for ScpLLM or StreamingLLMM Removing them has a significant impact on the perplexity of long texts. This conclusion is consis- tent with the paper (Xiao et al., 2024b).
Initial Tokens:
Following streamingLLMM, we conduct Positional Shifting for streaming applications, i.e, we focus on positions within the cache rather than those in the original text. Table 8 shows that this shifting plays a crucial role, as removing it significantly increases the perplexity (StreamingLLM increases from around 13 to over 400). noteworthy that SepLLM, even without employing this shifting, only sees perplexity increase around 200, which is much lower than StreamingLLM. This further underscores the separators' role for the stability in predicting tokens.
Positional Encoding's Shifting.
State-of-the-Ar Baselines. In addition, we compare SepLLM with multiple of-teo baseline methods, which demonstrates the effectiveness and simplicity of Se- PLLM. For detailed results, please refer to the Appendix E,
Separator Choices. A series of ablation studies are also conducted on the choice of separators. For SepLLM, whether it is the Fundamental Design or the Tailored Stream- ing Design, the choice of separator types serves as kind
Fiter-Interval Variant. Furthermore, to demonstrate the compression and summarization effects of separators on the information within the segments they divide, we propose
8

---

Accelerate LLMs by Compressing One Segment into One Separator
MMLU
represent the number of heads, hidden dimension in atten- tion layers, and the hidden dimension of
H,/H
r. r.KV (%)
n
humanity
social stem other overall
layers, respectively. Denote F as the class of continuous functions f : [0. 1] + Rdx" where d and n represent the dimen- sionality of input tokens and the sequence length, respec- tively. For any p 2 1, we use the ép distance to measure the difference between two continuous functions fı, /2 E FF de-
20/32
23.93
23.07 24.42 23.53 23.76
44.74
80
24/32
24.23
23.30 26.36 23.37 24.31
47.71
208
28/32
25.66
27.29
26.31 23.69
25.81
45.13
256
30/32
27.29
25.09
27.78 38.11
29.31
45.42
288
ScpLLM
57.66
76.21
56.49 72.19 64.68
44.61
256
1/p
Vanilla
60.49
76,50
56.61 72.19 65.72
100.00
ALL
fined as (Jo.qja.. (I - |2(x) dx The fol-
lowing theorem shows that the proposed SepLLM holds the universal approximation to arbitrarily suuuence-t--squenne continuous functions. For the complete proof and derivation process, please refer to the Appendices and K.
Table 9. Comparison with naive baseline method, where H, out of attention heads are set to sliding window attention, and the remaining heads use full attention. n represents the number of Neighboring Tokens (f.e., the sliding window size). r.KV (%) indicates the ratio of KV usage at runtime for the respective method compared to Vanilla.
Theorem 5.1. Given p > 1 and > 2, for any c > 0 and f E F, there exists SepLLM g E T such that dp (J.9) <.
fixed-interval variant, FiXLLM. In FixLLM, instead of attending to separator tokens, it attends to one token at fixed intervals, except for the Initial Tokens' and Neighboring Tokens' sections. We find that SepLLM significantly out- performs FiXLLM, which indicates that the compression and summarization capabilities of separators in SepLLMM cannot be replaced by fixed-interval tokens. Please see the Appendix for detailed results and discussions.
6. Native Sparse Attention and Language Modeling
SepL.M essentially models sparsity based on the natural semantic distribution of natural language. During the pre- training phase, SepLLM intentionally compresses segment information into the separator used to divide the segment. This approach closely aligns with the semantic distribution of natural language because the separator itself provides a division and summary of the current segment. The segments separated out are inherently semantically coherent, forming self-contained semantic units. Therefore, we can view Se- pLLM as a native sparse attention mechanism inherent to natural language itself. Moreover, SepLLM is suitable to replace StreamingLLLM (which only statically focuses on important positions and is overly sparse) as the fundamental baseline model for sparse attention mechanisms in LLMs.
4.8. Generalization and Information Retrieval
To verify the generalization of SepLLM, we adapt Se- pLLM to models of different architectures and scales. The results and discussions in Appendix D can validate the gen- eralization capability of our proposed SepLLM. Specifi- cally, we adapt ScpLLM to different backbones including Pythia-6.9B. Pythia-12B (Biderman et al., 2023), Llama-3- 8B-Ba --BBss//nsttuct (Dubey et al., 2024) and Falcon-40B (Al- mazrouei et al., 2023). Moreover, we also conduct the -eedle-in-a-Haaystack experiment, which further demon- strates the compression capability of separator tokens for segment information. As illustrated in Appendix F, Se- PLLM can retrieve the needle in most scenarios. In compar- ison, StreamingLLM (Xiao et al.. 2024b) cannot complete this task. In addition, we provide comparisons of SepLLM with more baseline models regarding mathematical reason- ing and logical analysis capabilities in the Appendix E. Fur- thermore, Appendices H and discuss in detail the effects and functions of separators from the perspective of imple- mentation logic.
7. Concluding Remarks
In this paper, we focus on efficient neural architecture modi- fication to address the computational and storage challenges in LLMs, especially when processing long inputs. From the visualization of attention maps, we find that certain separa- tor tokens consistently contribute high attention scores. In- spired by this, we propose SepLLM, new language model- ing perspective and a sparse attention mechanism, focusing attention computation on Initial, Neighboring, and Separator Tokens. To achieve wall-clock time acceleration, we also im- plement hardware-efficient kernels. Our training- free stud- ies suggest these separators effectively compress segment information, enabling efficient information retrieval. Unlike previous training-free methods, SepI.LMM can be incorpo- rated into the training phase, e.g., training trainingffrom-se -cracs or post-training, thus reducing disparities between training and inference. Extensive experiments across various settings have demonstrated SepL.LM's 's practical effectiveness.
5. Universal Approximation
In this section, we present the theoretical results on the universal approximation capabilities of encoder- based Se- PLLM. Details can be found in Appendices J and K.
H. be the class of SepL.LM, where H, dr, and dy
Let Sep

---

Accelerate LLMs by Compressing One Segment into One Separator
Impact Statement
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Chal- lenge. Preprint arXiv: weX.vv:183.... 1801.883..5555 2018.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training Verifiers to Solve Math Word Problems. Preprint arXiv:21 11111.1111... 2021.
References
Almazrouei, E.,, Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, É., Hesslow, D., Lau- nay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The Falcon Series of Open Language Models. Preprint arXiv:2311.1688 2023.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding, In North American Chapter of the Association for Computational Linguistics, 2019.
Beltagy, L, Peters, M., and Cohan, A. Long- former: The Long-Document Transformer. Preprint arXiv:2004.05150, 2020.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2020.
Biderman, S., Schoelkopf, H., Anthony, Q. Bradley, H., O'Brien, K., Hallahan, E., Khan, M., Purohit, S., Prashanth, U., Raff, E.,, Skowron, A., Sutawika, L., and Wal, O. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In International Conference on Machine Learning, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 Herd of Models. Preprint EiXiv22407221783 2024.
Bisk, Y., Zellers, R., Bras, R., Gao, J., and Choi, Y. PIQA: Reasoning about Physical Commonsense in Natural Lan- guage. In AAAI conference artificial intelligence, 2020.
Gao, L., Biderman, S., Black, S., Golding, L.,, Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. Preprint arXiv:2101.00021 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Ge, S., Zhang, Y., Liu, I.., Zhang, M., Han, J., and Gao, J. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. In International Conference on Learning Representations, 2024.
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Neural Information Processing Systems, 2020.
Geneva, N. and Zabaras, N. Transformers for Modeling Physical Systems. Neural Networks, 2022.
He, Z., Feng, G., Luo, S., Yang, K., Wang, L., Xu, J., Zhang, Z., Yang, H., and He, D. Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapola- tion. In International Conference on Machine Learning, 2024.
Chen, G., Xia, L., and Huang, C. Lightgnn: Simple graph neural network for recommendation. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM '25, 2025a. doi: 10. 1145/3701551.3703536
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding. In International Conference Learning Representations, 2021.
Chen, G., Xia, L., and Huang, C. Pre-training for rec- ommendation unlearning. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '25, 2025b. doi: 10. 10.114553726302.373000
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast Autoregressive Transform- ers with Linear Attention. In International Conference Machine Learning. 2020.
Chen, Y. Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. In International Conference on Learning Representations, 2024.
Li, J., Shi, H., Wu, S., Zheng, C., Li, Z., Jiang, X., Xu, H., and Jia, QuickLLaMA: Query-Aware Inference
10

---

Accelerate LLMs by Compressing One Segment into One Separator
Acceleration for Large Language Models. In Proceedings of the s1 International Conference on Computational Linguistics, 2025.
Wei, J., Wang, X., Schuurmans, D., Bosma, M.. Xia, F., Chi, E., Le, Q. Zhou, D., al. Chain-of-Though Prompting Elicits Reasoning in Large Language Models. In Neural Information Processing Systems, 2022.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. SnapKV: LLM Knows What You Are Looking for Before Generation. In Advances in Neural Information Processing Systems, 2024.
Welbi, J., Liu, N., and Gardner, M. Crowdsourcing Multiple Choice Science Questions. In Workshop on Noisy User- generated Text, 2017.
Xiao, C.,, Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. InfLLM: Training-Free Long- Context Extrapolation for LLMs with an Efficient Context Memory. In Neural Information Processing Systems, 2024a.
Liu, J., Cui, L., Liu, H., Huang, D., Wang. Y., and Zhang, Y. LogiQA: A Challenge Dataset for Machine Reading Comprchension with Logical Reasoning. In International Joint Conferences on Artificial Intelligence, 2021.
Merity, S., Xiong, C.,, Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. In International Conference Learning Representations, 2017.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Effi- cient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations, 2024b.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Feránddez, R. The LAMBADA dataset: Word prediction requiring broad discourse context. In Association for Computational Linguistics, 2016.
Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. PyramidInfer: Pyramid KV Cache Compression for High- throughput LLM Inference. Preprint arXiv:2405. 2024.
PyTorch. Flexattention:
The flexibility of pytorch with the performance of flashattention, 2024, URL https: //pytorch. org wwt.oorgg//ooooooooo/ooo/ogg/gge fllgg/fleatttettion.
Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native Sparse Atten- tion: Hardware- and Natively Trainable Sparse Attention. Preprint arXiv:2502.1 1089, 2025.
Rae, J., Potapenko, A., Jayakumar, S., Hillier, C., and Lil- licrap, T. Compressive Transformers for Lang-Range Sequence Modelling. In International Conference on Learning Representations, 2020.
Yun, C., Chang, Y.-W., Bhojanapalli, S., Rawat, A. S., Reddi, S., and Kumar, S. O(n) connections are expressive enough: Universal approximability of sparse transform- ers. Advances in Neural Information Processing Systems, 33:13783-13794, 2020.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. Journal of Machine Learning Research, 2020.
Zaheer, M., Guruganesh, G., Dubey, K., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big Bird: Transformers for Longer Sequences. In Neural Information Processing Systems, 2020.
Schlag, I., Iric, K., and Schmidhuber, J. Linear transform- ers are secretly fast weight programmers. In Interna- tional Conference on Machine Learning. pp. 9355-9366. PMLR, 2021.
Zhang, J., Zhao, Y., Saleh, M., and Liu, P. PEGASUS: Pre- training with Extracted Gap-sentences for Abstractive Summarization. In International Conference on Machine Learning, 2020.
Shi, H., Gao, J., Ren, X., Xu, H., Liang, X., Li, Z., and Kwok, J. SparseBERT: Rethinking the Importance Anal- ysis in Self-attention. In International Conference on Machine Learning, 2021.
Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. Preprint arXiv:2406.02066 2024.
Su. J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L.., Cai, R., Song, Z., Tian, Y., Ré, C.,, Barrett, C., et al. H₂O: Heavy- Hitter Oracle for Efficient Generative Inference of Large Language Models. In Neural Information Processing Systems, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L.,, and Polosukhin, I. Attention is All you Need. In Neural Information Processing Systems, 2017.

---

Accelerate LLMs by Compressing One Segment into One Separator
Zheng, C., Gao, Y., Chen, G., Shi, H., Xiong, J., Ren, X., Huang, C., Jiang, X., Li, Z.,, and Li, Y. Self-adjust softmax. Technical report, 2025.
Vanilla
SepL.LM
SepLI.M
(Full Attention)
(n=64)
(m=128)
time per iteration (ms)
2524.45
1648. 11
1653.11
samples second
405.82
622.31
620.30
Zhu, Q. Duan, J., Chen, C., Liu, S., Li, X., Feng. G., Lv, X., Cao, H., Xiao, C., Zhang, X., et al. SampleAttention: Near-Lossless Acceleration of Long Context LLM Infer- ence with Adaptive Structured Sparse Attention. Preprint arXiv:2406.15420 2024.
Table 10. The details about training acceleration.
We can see that the speeds of SepLLMM(=1288) and Se- (LLM(n=64) are almost the same, which attributed to the excellent parallelization capability of Sep-Artention module.
Appendix
D. The Performance of Different Models
Different Architectures. Concerning different decoder- only models, we test our SepLLM on Llama3 and Pythia backbones on PG19 test dataset (generating 64K tokens).
A. Visualization of Attention Scores
We take Llama-3-8B-instuue (Dubey et al., 2024) as the model for visualization. The input sentence "Natalia sold clips to 48 of her friends in April, and then she sold half many clips in May. How many clips did Natalia sell altogether in April and May? Answer Natalia sold 48 clips in April. In May. she sold half as many clips as she did in April, so she sold 48/2-24 clips in May. Therefore, Natalia sold a total of 48+24=72 clips in April and May. The answer 72 and the visualization of different attention maps are shown in Figure 12,13,14.
The results are shown in Table (a=4, c=800 for both Se-
pLLM and StrmLLM. s=64.==256 for SepLLM).
Backhone
Arch.
r.KV
ppl
time(s)
Vanilla
64K
32K
1037.6
4160.7
Pythia-6.9P
StrmLLM
800
800
15.9
1522.6
ScpLLM
800
562
15,8
1456.0
Vanilla
64K
32K
1090.8
3380.6
Llama-3- 3-BB
StrmLLM
800
800
37.9
1096.0
SepLLM
800
562
33.4
1049.7
Table 11. The comparison of SepLLM adapted to differrnn archi- tectures.
B. The Evolution of KV Caches
To explain the dynamic design for streaming setting better, we illustrate the detailed evolution of KV caches in Figure 7. As can be seen, n and Sizerun are both periodic functions after mo tokens. And the average KV cache usage is much less than the maximum capacity c.
From the above table, it can be seen that for models with similar size, setting similar KV retention rate can yield similarly good performance.
Different Scales. To learn the generalization to different scales, test SepLLM Pythia-6.99 and Pythia- 12B backbones on PG19 test dataset (generating 20K tokens). The results are illustrated in Table 12.
Backbone
s
w
r.KV
ppl
time(s)
64
256
800
562
13.0
445.0
Pythia-6.9B
4
64
800
1024
946
12.7
450.4
me0
64
928
1280
1138
12.7
454.4
Sizr.an
Pythia- 12B
4
64
256
800
562
12.1
577.0
Table 12. The comparison of SepLLM adapted to Pythia (Bider- man et al., 2023) with different scales.
w
Compared to Pythia- 12B, the smaller model Pythia-6.99 will have a higher perplexity the entire KV cache capacity is the same (c=800). Therefore, it is necessary increase to achieve a lower perplexity close to that of Pythia- (c=800), On the other hand, the larger models will have lower perplexity but require a longer inference time.
Figure 7. The evolution of KV caches in the streaming setting.
C. Training Acceleration
We list the detailed wall-clock time per iteration and through- put Table 10. The speed-up ratio around 1.53.
Larger Models Falcon-40B (Almazrouei et al.. 2023) is another larger architecture we adapted evaluate
12

---

Accelerate LLMs by Compressing One Segment into One Separator
the scalability of our proposed SepLLM. The exper- iment results shown in Table 13, where set a=4,5-64,ww5122720,, c=800/1024 for SepLLM, and a=4, c=800/1024 for StreamingLLM. And the conclusions are similar to the previous parts.
flexibleexttract
strict-match
r.KV (%)
Vamilla
77.79
77.26
100.00
ScpLLM
77.18
77.18
47.36
H20
76.27
75.06
47.54
SnapKV 76.50
73.62
47.54
PyramidKV 75.82
72.02
47.54
Length
Methods
r.
ppl
time (s)
StrmLLM
1024
1024
8,98
1512.88
Table 15. Evaluation results and average runtime KV cache usage for experiments on GSM8K-CoT with 8-shots, compared to multi- ple baseline methods.
StrmLLM
800
800
9.02
1430.59
20K
SepLLM
1024
906
8.92
1440.89
SepLLM
800
690
9.00
1368.07
StrmLLM
1024
1024
11.01
4844.79
StrmLLM
800
800
11.09
4623.90
PyramidKV (Zhang et al., 2024). All methods are config- ured to retain nearly identical runtime KV cache usage. The results are shown in Table 15. It can be observed that Se- pLLM, without requiring complex importance evaluation mechanisms, is able to maintain strong reasoning capabili- ties simply by compressing segment information.
64K
SepLLM
1024
906
10.96
4619.63
ScpLLM
800
690
11.07
4414.72
Table 13. The comparison of SepLLM adapted to Falcon-40B (Al- mazrouei et al., 2023).
F. Needle in a Haystack
Base Instruct. In general, whether it is the base model or the instruction-tuned model, we can condense the seg- ment information into the corresponding Key-Value pairs of the separator tokens. To illustrate, we fine-tune Llama-3-8B- instruct and Llama-3-8B 3-8-bas models (Dubey et al., 2024) on LongAlpaca dataset (Chen et al., 2024) for only 200 and 500 steps, respectively. After fine-tuning, we take GSM8K- CoT (Cobbe et al., 2021) as the benchmark for reasoning ability evaluation. We find that both base and instruction- tuned models exhibit excellent performance (matching or surpassing the vanilla model with original attention mech- anism). The only difference is that for -lama--3-BB-base we need to fine-tune for more steps to achieve such perfor- requires fewer
To evaluate the long-context information retrieval ability of our proposed SepLLM, we take Needle in a Haystack2 as the benchmark and compare the performance of Se- pLLM and StreamingLLMM The results are shown in Fig- ure 8,9,10, 9,10011 11 and SepLLMM can achieve more scores com- pared to StreamingLLM, This experiment indeed validates that SepLLM can effectively compress information of seg- ments into the KV corresponding to separators, as even though the KV corresponding to tokens in the needle (except for possibly existing separators) are discarded, SepLLM can still retrieve the needle.
mance. In comparison, fine-tuning steps. Even in training-free scenario, Llama- 3-8B-instrue -0B-itttuuct demonstrates decent performance. This indi- cates that the embeddings of Llama-3--88 11-8--iinstect align with the distribution required by the SepLLM architecture better.
G. Choice of Separators
In this paper, we treat the choice of separators as a hyper- parameter. Since the number of commonly used separa- tors typically limited in most languages, we recommend including all types of separators as much as possible. In Table 16, we present the experimental results based on differ- ent choices of separators. We use the GSM8K-Cof metric to reflect the performance differences caused by these choices (GSM8K-CoT is a commonly used metric for evaluating the mathematical reasoning and logical analysis capabilities of LLMs). It can be observed that when using four com- mon separators instead of the default nine, the performance shows certain degree of decline. Furthermore, when only " and "?" are used, the model's reasoning ability drops significantly. When all separators are removed, this perfor- mance degradation becomes even more pronounced (i.e., StrmLLMmn=256), While increasing the number of Neigh- boring Tokens (n) to ensure the total amount of retained KV cache remains unchanged can partially mitigate the per-
Backbone
Algorithm
GSMSK-CO
r.KV (%)
Base
Vanilla
54.44
100
SepLLM ft.
55.95
47.36
Instruct
Vanilla
77.26
100
SepLLM ft.
77.63
47.36
Table 14. The comparison of SepLLM adapted to Llama-3- 8B (Dubey et al., 2024) of base or instruct versions.
E. Extended Comparisons
We use GSM8K-Co (Cobbe et al., 2021), the most com- monly used metric for testing mathematical reasoning and logical analysis, to compare SepLLM (a=3, n=256) with other state-of-the-an training-fret methods, including H2O (Zhang et al., 2023), SnapKV (Li al., 2024), and
fhtts:/ggihabbcom/ggaaaaddLLLLL A
13

---

Accelerate LLMs by Compressing One Segment into One Separator
formance loss (i.e., StmmLLM(==380)) it falls far short of restoring the performance to the levels achieved with the default nine separators or the four common separators.
the two variants using the GSM8K-CoT (Cobbe et al., 2021) and MMLU (Hendryeks al., 2021)) benchmarks under training-free setting, based on the L backbone. The results in Table 17 show that LixLLM has a significant gap compared to SepI.I. in both mathematical logical reasoning and knowledge- ties. Therefore, the summarization and compression effects of separators in SepLLM for the segments they divide can- not be replaced by fixed-interval tokens.
H. Discussions on Separators
reasoning capabili-
We provide the following assumptions and discussions on why keeping the KV corresponding to separators can main- tain the performance of the original model.
Training from Scratch During our training process, we enforce that each current token can only see its preceding neighboring tokens, separator tokens, and initial tokens such that the model is compelled to condense the information of each segment into the Key-Valux pairs corresponding to the separators through the self-attention mechanism. Therefore, the hidden embedding of a separator is functionally similar to the state space of an RNN, even though the computa- tion method differs as we utilize the attention mechanism. Furthermore, since the length of each segment is typically finite, short and balanced (He et al., 2024), the compressed information is efficient and less likely to be forgotten.
J. Universal Approximation
In this section, we theoretically analyze the universal ap- proximation capabilities of encoder-based SepLLM. Let Topp H.dd.dd be the class of SepLLM, where H, da, and dy represent the number of heads, hidden dimension in atten- tion layers, and the hidden dimension of feed-forward layers, respectively. In the attention layer of SepL.MM, token can attend to its neighboring tokens with a sliding window of size l (i.e., tokens in the range ii f, i + E)) and all special tokens of the sequence. Additionally, we assume that for at most s successive tokens, a special token will appear in the sequence. Denote F as the class of continuous functions f : [0. 1jdx= - Rddn, where d and n represent the dimen- sionality of input tokens and the sequence length, respec- tively. For any p 2 1, we use the lp distance to measure the difference between two continuous functions, f1. f2 E F, de- 1/p The fol- lowing theorem shows that the proposed SepL.MM holds the universal approximation arbitrarily sequence-to-sequeee continuous functions.
Training-Free First, these separators (commas, periods) are extremely high-frequency tokens (both in the pre- training text and in the text generated by the language model). Thus, during the pre-training process, these sep- arators are the most common context for all other tokens in the vocabulary. Consequently, their embeddings exhibit greater similarity, resulting in larger attention values when multiplied with other tokens. Furthermore, from a logical standpoint, it is evident that separators need to be generated by the language model very frequently. Therefore, their at- tention values with respect any other token cannot be too small; otherwise, they would not be generated frequently by the language model. Tokens with larger mutual attention val- ues will incorporate more information from the other tokens. Hence, from a semantic perspective, generating a separator serves as a summarization of the current segment, naturally dividing and summarizing the contextual semantics.
fined as (Fo.jjax. (x) - (I(X) dx
Theorem J.1. Given p 1 and n > 2, for any C >> O and ʃ E F, there exists SepLLM y E T2p. such that dp (f.g) <.
We outline the key steps of the proof here, with the full details provided later. The proof follows the approach in Yun et al. (2020).
We begin by dividing the region [0, 1]dx= into set of grid points Gs = (0, 26, ..... 1)dxn, where each point in [0, 1jdxn corresponds to a cube defined by these grid points. Here, 5 >> 0 determines the resolution of grid points. Specifically, for any X E [o, 1jdxn, there exists a grid point X, E Gs. such that X lies within the cube X8 + [0, ddxn, We then assign the same function values to all inputs belonging to the same cube, as determined by a piecewise constant function f. For any 0, there exists sufficiently small 8 > O. such that dp. < 1
Step 1:
I. Fixed-Interval Variant
Sparsity is ubiquitous across various types of neural net- works (Shi et al., 2021; Chen et al., 2025a; Zheng et al., 2025; Yuan et al., 2025; Chen et al., 2025b). To verify the sparsity caused by separators as natural semantic di- visions in natural language and the resulting importance of these separators, we propose another variant, i.e., when calculating sparse attention, instead of focusing only Separator Tokens between the Initial Tokens and Neighbor- ing Tokens, we attend to one token at fixed intervals (c.g., every 8 tokens or 16 tokens), while the other tokens are masked, namely FiXLLM. We evaluated the mathematical reasoning ability and knowledge-basse based reasoning ability of
Step 2: attention layers and feed- forward layers of SepLLM by the hardmax operator (i.e., arg max) and piecewise linear functions (at most three pieces). We denote the class of
We replace the softmax and ReLU activation in
14

---

Accelerate LLMs by Compressing One Segment into One Separator
M (n=256)
SepLLM (n=256)
SepLLM (n=256) StrmLLM (n=256) StrmLLM (n=380)
Separators
41 u. ugu upn u,n o.n
36.44 6..9 uQ"
...1 ug"
None
None
r.KV (%)
47.36
37.92
31.92
47.54
exxtableexttract
77.18
76.68
70.66
68.84
71.42
strict-match
77.18
76.85
70.36
67.63
70.89
Table 16. Evaluation results and average runtime cache usage for training-free free experiments on GSM8K-COn with 8-shots, based on different choices of separators.
GSMSK-CoT
MMLU
flexible
strict
r.KV (%)
humanities
stem
social
other
overall
r.KV (%)
Vanilla
77.79
77.26
100.00
60.49
56,61
76.50
72.19
65.72
100.00
FiXLLM (A=5, #=256)
70.43
70.43
45.64
55.52
54.80
72.99
69.75
62.33
50.20
FiXLLM (A=4, n=256)
72.71
72.33
49.08
55.92
54.39
74.36
70.81
62.91
53.32
SepLLM (#=256)
77.18
77.18
47.36
57.66
56.49
76.21
72.19
64.68
44.61
Table 17. Evaluation results and average runtime KV cache usage for training- free experiments on GSM8K-COL 8-shots and MMLU 5-shots. For SepLLM and FixLLM, three initial tokens' KV are kept. A denotes the interval size for FixLLM and is the number of retained neighboring tokens' KV. r.KV (%) represents the ratio KV usage at runtime for the respective method compared to Vanilla.
the modified SepLLM models by T HH.d For the above piecewise linear function f, there exists modified SepLLM g E Scp such that f.
quantization mapping 9T"
eet (t) = kō, ikō < 11 (k+
k E (0: n/ō 1],
otherwise.
Step 3: Finally, approximate the modified SepLLM E Scp by standard SepLLM g T i.e. we have (). < 5. This approximation is justified by the fact that the softmax function can approximate the hardmax op- erator arbitrarily closely when the temperature parameter is sufficiently large. Additionally, feed-forward networks with ReLU activation can effectively represent any piecewise linear function.
(5)
There exists a modified SepLLM with feed-foeooorde feed-forward layers and identity atlention layers, 94 E - <Tspp composed of nd "6 layers with n, = 1 and piecewise linear activation func- tions, such that 9g realizes the quantization mapping S,"
Let Z 9 E) denote the quantized input token matrix, which corresponds to mapping X + E to the grid point at the leftmost corner of the cube. Denote Go as the set of all quantized token matrices derived from X + E. The following definition of contextual mapping aims at uniquely distinguishing each grid point (i.e., quantized tokens) from all possible combinations.
K. Proof for Theorem J.11
Lemma K.1 (Lemma 5 in Yun et al. (2020)). For any f E F, > O, and p 2 1, there exists a piecewise constant function f, such that ff <
Definition K.3 (Contextual Mapping). For given set of grid points G3 C Rdxn, contextual mapping Gs + R" satisfies:
To identify the position of tokens in SepLLM, we include the position encoding E E Rd into the token X, i.e., the input token X + E. Here, for theoretical convenience, the positional encoding matrix defined as
For any E Go. all entries in ( are distinct.
For any G, G' E Gs with G + G¹, all entries of (g(G), ((G') E Rm are distinet.
E = [n 101,0,1 ................................ (n 2)]].
Lemma K.4. There exists a function ge E T2.1111 2,111 Bdxx + Ædxa, consisting of nnp + +[T layers of modified SepLLM (purely attention layers with identity feed forward layers), such that qi (Z) u ge ge is contextual mapping. Here, the modified SepLLM means replacing all softmax functions with hardmax operators.
With this encoding, the input token lies within the range X + takes in the range [0, nj)dn. The input token is then mapped to grid points using a quantization function Ig. which be implemented using feed-forward neural networks with piecewise linear activation functions (i.e., the feed-forward layers of modified SepLLM).
Proof. Denote i-th column of Z as Z, and let (1,8-1 5-2 ...... 5 f-d+1) Based on the definition of the
Lemma K.2 (Lemma 6 in Yun et al. (2020)). Consider the
15

---

Accelerate LLMs by Compressing One Segment into One Separator
quantization mapping By and the selection of positional en- coding matrix E, we have
matrix 2), resulting in:
Za = Z3 + 80 - 2a)e1,
u
TT Z1 E
and
d-1
d-1 Σ
23 := u Z3 23 + (2) 22)0 22-
(n- 11-1] 1)
:8: (n- 1)
8 +
d+1
-8
We repeat this process for the remaining tokens until the last token k = n. As a result, the obtained token matrix Z satisfies:
i=0
i=0
u TT Zk E
d-1
d-1
(k- 2)
Σ ==0
8- :8 (k- 2)
Σ 1=0
8 + 8-
31 22
Enp
where ok == T Zz. In summary, there exists a modified SepLLM with n-1 layers capable of transforming the token 3d matrix Z into Z.
(6)
for k E 12 ing to distinct are disjoint. Denote u. T Zg as Zk. it follows that 22 < 23 ... on < ø1. We define the operator
: nd.. We observe that those intervals correspond-
Now, we prove that the mapping from Z to ž, is injective. Suppose there exist two token matrices Z, Z' E Go. such that in = z/h By induction, we have
((Zb)R
u² ZH [1' (u²ZA,))
ZAm) )" (u"
Zz- []]
nn 2n + 8 (En 2n)
maxxEAA, u² mingeA u Z3
if uf Z > b,
_...
if T Z < b,
2n + Σ 8 id (2n-i 2n+1-i)-
where As denotes the set of tokens that the token & can at- tend to in SepLLM, consisting of its neighboring tokens and special tokens. The operator Vr ( b)& can be implemented using a one-head attention layer in the modified SepLLM with the hardmax operator the activation function. We further define the following operator, which can be imple- mented using a two-head attention layer in the modified SepLLM:
If In F 2/g Izn - 2/11 S 5 5-d+1 - 6. However, the term Σ 8 5-id (idddd(nn (2n-i 2n+1-i) is dominant, and cannot cancel In =/1-- If nn = z' but 2n-1 + 2 we encounter similar contradiction. Since each term in the sum has a different scale of 8-1, ānn = ž holds if and only if Z = Z'.
For the current token matrix - 2 Z dominant term for each column is located in the first row, where
Zn the
& ( Z c; bmin: bmax)x
Z + (M (Z bmaxx) V (Z bmin))) ei
(Zk)
(Zk), +5
2k).
Z e (maxjEAL u² Z, minjeA. u² Zj) el
z
Moreover, the dominant term for the first element of each column is 8-1Ex-1- Since ž, can be viewed as the iden- tity of the original token matrix Z, the last column, which depends on Σ,, is distinct for different token matrices. How- ever, for the other columns dominated by E5 (k + n), the uniqueness is not guaranteed. To address this, we apply a series of token transmission steps, propagating information from the last token to all other tokens. The core idea relies on the fact that the last token can attend to the nearest spe- cial tokens with the help of neighboring tokens, requiring at most [ layers. Following Section E.2.4 in (Yun et al., 2020), after [ layers, žn successfully copied to all to- kens. As result, each column of the updated token matrix,
Here, the first condition is satisfied if bmin < uT Zs < Dmax: and the second condition applies otherwise. For l (the second token), we apply the operator 4(Z; 8-44 d, b + 2) for 8 times, with b varying in the range [0 : 8 5-d+1 - 5). Note that the operator modifies only the k-th token while leaving all other tokens unchanged, because u1 Z% lies in disjoint intervals. After this operation, we denote the updated second token as Zz, and we have
7 = Zz + 8 d21 - -22)e1,
and
82 u T Z 22 + (21 22)8 > 21.
denoted Z, depends on Zn. Furthermore, all elements of Z are distinct and lie in disjoint intervals. Since än acts as an identity to the token matrix Z, the mapping u Z satisfies all conditions of contextual mappings. The map- ping from Z to Z can be realized by a-1 + [[] layers of a
Similarly, Similarly, for the third token (k- = 3), we apply the operator o( 8-d: b- 2. b+b++ ) with varying in the range [X:10 8 --- 8 + 8-1+1 - 6) This op- eration modifies only the third token (the third column of
16

---

Accelerate LLMs by Compressing One Segment into One Separator
modified SepLLM, consisting of purely attention layers and identity feed-forward layers.
SepLLM. Here g represents the modified SepLLM with of + [ an layers. 'hee following lemma states that the modified SepLLM can be approximated by standard SepLLM with arbitrary small error.
m
Lemma K.5 (Lemma 8 in Yun al. (2020)). For the contex- tual mapping 9e in Lemma K.4, there exist zēn -layer modi- fied SepLLM (composed of feed-forward layers and identity attention layers), denoted as E Tsp such that
Lemma K.6 (Lemma 4 in Yun et al. (2020)). For any mod- ified SepLLM 9 E T and any > 0, there exists SepLLM E Tee such that dp(g. g) <.
221111
RR - Rdxx
go (g)(Z)) f(z E),,
By combining all the lemmas above, we conclude that for any f F and > 0, there exists a SepLLM g E T such that dp(f.9) .) <.
where k denotes the k-th colunn of the token matrix.
From the above analysis, obtain
g X =
(X + E) = f(X),
L. AI Assistant
where the quantization mapping 9q. the contextual mapping Ie and the value mapping Iu are all realized by the modified
We only use Al assistants (e.g., SPT-3.5-Turbbe to polish and refine the article.
17

---

Accelerate LLMs by Compressing One Segment into One Separator
sensiua mEV Entreat Lang Hanyttarkk
20
2..
210
11CC
122
120
S4I
100
2144
1408
142
77E
2300
Ban Len2
Figure 8. Needle in-a a-Hassttckk test results for Streaming LM (n=64) based on Pythia- 1iiee-1000----- 60M-dedupee..
Prequre beeing Pat Herrr---eeededeee gndd wis 72648
4.0
110
13.
2010
Figure Needle- in-a- H-y-sttac test results our M H/T) based on Pythia- 60dddddeeed.
Der netinal fros Contect Le wegi Pheedaa P.A. lleySlack"
2.0
1.10
21.0
33.0
440
47.0
Figure 10. Needle-in in-a- mn-a-Hayaa a-Hasstack test results for our ScpLLM first/last 2 layers (4 layers in total): full attention) based on Llama-3-8Ba 3-8-innttrcc... initial tokens are kept.
Dart nerrinal fareed Contrct Leng Praeeda litesiiaca.
2.0
200
33.0
440
47.
Figure 11. -eedltt-nn-a-Haayttack test results for SepLLMMn=20088; first/last layers layers in total): full attention) based Llama-3-8B-ines 3-B--istrucc... 32 initial tokens are kept.
18

---

Accelerate LLMs by Compressing One Segment into One Separator
Layer and Head o
varge encrreer yprr yy 1o
Figure 12. example of attention map in Llama-3-8B- 3-8t-Insttuct (Layer 0 Head
19

---

Accelerate LLMs by Compressing One Segment into One Separator
Layer and Head 0
0,0
900 8 "p 4 Waywery 40 say
Figure 13. An example of attention map in Llama-3-8B- Bamma-3-B-IInstruee (Layer I and Head 0).
20

---

Accelerate LLMs by Compressing One Segment into One Separator
Layer 2 and Head
0.0
-ayg x 19r y "ggls Wayoery 40 my
Figure 14. An example of attention map in Llama-3-8B- 8-----Isstttt (Layer 2 and Head 0).
21


Reviews:
[{"review_id": "eWJ7ER4FQv", "reviewer": "Reviewer_pjB2", "summary": "The paper introduces SepLLM, a novel framework aimed at improving the efficiency of large language models (LLMs) by leveraging the observation that separator tokens (e.g., commas, periods) disproportionately contribute to attention scores. The authors propose compressing segment information into these separator tokens, reducing computational costs and memory usage while maintaining model performance.\n\n## update after rebuttal\n\nThe author's rebuttal addressed most of my concerns; I will keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n**Should there be any need for further clarification to assist in advancing our scores, please do not hesitate to inform us. Thank you very much.**\n\n> Q1. important evaluations such as long ppl and \"needle in a haystack\"\n\n**A1**.\nWe have provided numerous results on perplexity (ppl) for extremely long test datasets. For instance, based on the Llama3-8B model and the PG19 test set, we conducted inference on extremely long sequences of up to 4 million tokens. The results are presented in Table 4 of our paper, which we have copied here for your reference.\n\n| Input Length |3M |3.5M | 4M |\n|----|----|----|----|\n|StreamingLLM| 36.4|35.8|36.1|\n| SepLLM(s=32)|34.9|34.2|34.5|\n|SepLLM(s=64)|34.3|33.7|33.9|\n\nAdditionally, we have also conducted experiments on **Needle In A Haystack**, with results and detailed discussions presented in Figure 8-11 of our submitted paper. The experimental results show the long-context retrivie capacity of SepLLM.\n\n> Q2. retained one token for every fixed number of tokens\n\n**A2**.\nAs suggested, we conducted such experiments and the results are shown in the table below. \n\n| | GSM8K-CoT | | | MMLU |  |  |  | |   |\n|---|---|---|---|---|---|---|---|---|---|\n| | flexible | strict | r.KV(%) | humanities | stem | social | other | Overall |  r.KV(%) |\n|Vanilla |77.79|77.26|100.00|60.49|56.61|76.50|72.19|65.72|100.00|\n| FixLLM($\\Delta$=5,$n$=256)|70.43|70.43|45.64|55.52|54.80|72.99|69.75|62.33|50.20|\n| FixLLM($\\Delta$=4,$n$=256)|72.71|72.33|49.08|55.92|54.39|74.36|70.81|62.91|53.32|\n|SepLLM($n$=256)| 77.18| 77.18| 47.36| 57.66| 56.49| 76.21| 72.19| 64.68| 44.61|\n\nHere, FixLLM represents the variant you mentioned, where $n$ still denotes the number of neighboring tokens, and every $\\Delta$-th tokens are also remained outside the neighboring window. Experimental results indicate that FixLLM is still inferior to SepLLM. \n\n\n> Q3.  generalization issue as separators may vary across different languages or datasets. \n\n**A3**.\nIn our perspectives, separators are general across different languages or datasets. In modern commonly used languages, separators are widely present (including but not limited to English, German, French, Italian, Chinese, and etc, covering over 7 Billion people). While most languages use separators similar to modern English, a few languages still retain the usage of their traditional separators. For example, in Bengali, the period (\".\") is commonly used as a sentence-ending marker, but the traditional sentence-ending marker \"।\" (vertical line) is still in use. During training and inference, we only need to specify the separators we want to use, and the commonly used separators in most languages are around 10 in number. \n\nEmprically, we have verified the robustness of our SepLLM across various tasks or datasets, including GSM8K, MMLU, ARC, PIQA and so on. All emprical results can demonstrate the generalization of SepLLM as well.\n\n> Q4. low entropy within chunks and high entropy between chunks\n\n**A4**.\nWe agree that entropy-based grouping is a feasiable apporach. However, entropy-based grouping is not training/inference efficient since we need to calculate the attention mask by some algorithms. In comparison, SepLLM follows the natural modeling of the language and split the segments by separators, which is more efficient and more robust.\n\n> Q5.  some heads are converted to sliding window attention while allocating the remaining computation to a full window attention? This would represent a standard hybrid baseline.\n\n**A5**.\nAs suggested, we added the following experiments.\n\n| | MMLU | | | | |r.KV(%) | n|\n|---|---|---|---|---|---|---|---|\n| #Sparse_Heads/#Total_Heads | humanities | social sciences | stem | other |overall |\t| |\n|20/32 | 23.93 | 23.07 | 24.42 | 23.53 | 23.76|44.74|80|\n|24/32 | 24.23 | 23.3 | 26.36 | 23.37 | 24.31|47.71|208|\n|28/32 | 25.66 | 27.29 | 26.31 | 23.69 | 25.81|45.13|256|\n|30/32 | 27.29 | 25.09 | 27.78 | 38.11 | 29.31|45.42|288|\n|SepLLM\t|57.66|76.21| 56.49| 72.19|64.68|44.61|256|\n|Vanilla | 60.49| 76.50 |  56.61 | 72.19 | 65.72|100||\n\nFor example, \"20/32\" means there are 20 sliding-window heads out of a total of 32 heads (the remaining heads using full attention).\nAs can be seen, with similar amount of KV cache, SepLLM is much better than these hybrid baselines.\n\n> Q6. any methods to further improve the KV compression rate of SepLLM\n\n**A6**.\nOne possible method may be the multi-level version of SepLLM. In the current version, we compress the \"segment-level\" into general seperators. In the multi-level version, we may define \"sentence-level\" or \"paragraph-level\" as a higher-level seperation and compress a set of previous seperators into one more condensed KV Cache. Then, the KV compression rate will be improved.", "claims_and_evidence": "Yes, the authors propose a sparse attention mechanism based on separator tokens and validate its effectiveness through experiments.", "methods_and_evaluation_criteria": "Yes, the authors first observe the attention matrix and find that separator tokens have stronger attention weights, which leads to the proposal of the SepLLM method. And the evaluation on long-context ppl is the standard setting in sparse attention.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Although there are relatively sufficient experiments on long texts, some important evaluations are still missing, such as long ppl [1], because traditional ppl evaluations may not accurately reflect long-text capabilities on truly extended texts. Additionally, testing tasks like \"needle in a haystack\" would help assess long-text capabilities.\n- Furthermore, some simple baselines were not considered. For instance, while this method retains separator tokens, what would happen if we retained one token for every fixed number of tokens? Especially after some training, fixed sparse strategies might also prove to be quite efficient.\n\n[1] What is Wrong with Perplexity for Long-context Language Modeling?", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to prior research on sparse attention mechanisms and efficient memory management in large language models (LLMs). The observation that separator tokens receive disproportionately high attention aligns with earlier findings on the importance of specific tokens (e.g., attention sinks) in long-context modeling, as highlighted in works like StreamingLLM and SnapKV. Additionally, the proposed SepLLM framework builds on ideas from sparse attention methods, such as BigBird and Longformer, but differentiates itself by focusing on data-dependent token selection (separator tokens) rather than fixed patterns.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The reliance on separator-based design might affect generalization, as separators may vary across different languages or datasets. A more general consideration could involve first performing adaptive chunking on the data, ensuring low entropy within chunks and high entropy between chunks, followed by compressing each chunk.", "other_comments_or_suggestions": "None", "questions_for_authors": "- What would happen if sparse attention is not used, and instead some heads are converted to sliding window attention while allocating the remaining computation to a full window attention? This would represent a standard hybrid baseline.\n- Are there any methods to further improve the KV compression rate of SepLLM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrnjJo7koE", "reviewer": "Reviewer_Crr7", "summary": "## update after rebuttal\nThe author's rebuttal addressed many of my concerns; however, I am still a little hesitant about the generalisability. I will change my score from 2 to 3.\n\n\n* The paper identifies a key pattern: certain seemingly meaningless special tokens (i.e., separators) contribute disproportionately to attention scores compared to semantically meaningful tokens.\n* Plug-and-play method for compressing such special tokens and removing redundancy.\n* Also implement efficient kernels for training acceleration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n**Should there be any need for further clarification to assist in advancing our scores, please do not hesitate to inform us. Thank you very much.**\n\n>  Q1. \"Our positional encoding strategy for streaming settings is the same as the state-of-the-art StreamingLLM\" more information on the exact technique would have been helpful, as the correct positional encoding seems to inspire the design a lot. Is the technique used the same as ROPE? Better?\n\n**A1**.\nWe just adopt RoPE. However, in scenarios involving infinitely long streaming input, we follow StreamingLLM[1] and apply PE's shifting; i.e., we focus on positions within the cache instead of those in the original text. For instance, if the current cache contains tokens [0, 1, 2, 3, 6, 7, 8] and is in the process of decoding the 9th token, the positions assigned are [0, 1, 2, 3, 4, 5, 6, 7], rather than the positions in the original text, which would be [0, 1, 2, 3, 6, 7, 8, 9]. However, for training and downstream tasks of general length (<4K; Sec 3.1), we simply use standard RoPE without PE's shifting. See **Sec. 3.3;4.6 and Tab. 8** for details.\n\n> Q2. Ablation studies show the importance of additional tokens, but it seems a better idea would have been to show the effect of separator tokens as that is the main claim for improvement.\n\n**A2**.\nIndeed, the ablation studies you propose are illustrated in **Tab.1-8 and Fig. 5** (for both training & training-free). Without separators, SepLLM degrades to StreamingLLM. As demonstrated by the experimental results across so many settings and tasks, the separators indeed provide significant benefits to the architecture design. See more studies on separators in **A6 to Reviewer kB2E**.\n\n> Q3. How does SepLLM compare against well known KVCache compression techniques such as H2O and SnapKV?\n\n**A3**.\nAs suggested, we add the H2O, SnapKV and PyramidKV [2] as baselines and the results on GSM8K_CoT and the challenging MATH are as follows (All based on Llama3-8B-Inst backbones).\n| | GSM8K-CoT | | |\n|---|---|---|---|\n| | flexible | strict | r.KV(%) |\n| Vanilla |77.79| 77.26| 100.00|\n| SepLLM | 77.18 |77.18| 47.36|\n| H2O | 76.27 |75.06| 47.54|\n| SnapKV | 76.5 |73.62| 47.54|\n| PyramidKV | 75.82 |72.02| 47.54|\n\n| | MATH | | | | | | | | |\n|---|---|---|---|---|---|---|---|---|---|\n| |**overall** | **r.KV(%)** |algebra | counting_and_prob | geometry |intermediate_algebra | num_theory | prealgebra | precalc | \n| Vanilla |27.32| 100.00|37.83| 24.26|18.58|12.62|21.11| 48.34|11.72|\n| SepLLM | 26.74 |32.4 | 37.99 | 24.05 | 19.83 |10.52|18.7|48.56|10.62 |\n| H$_2$O | 23.72 | 33.1| 34.71 | 17.72 | 15.03 | 11.41 | 19.63 | 41.79 | 8.24|\n| SnapKV | 25.68 | 33.1 | 37.49 | 21.94 | 16.91 | 10.41 | 20.37 | 45.58 |9.71|\n| PyramidKV | 25.1 | 33.1| 37.49|20.46| 16.49 | 9.97 |19.44 |44.55 |9.34|\n\nWith similar amount of KV cache, SepLLM can achieve better performance.\n\nH2O and SnapKV are training-free methods that rely on full attention for pretraining and prefilled KV, but their inference-stage KV selection causes training-inference inconsistency and query dependency. SepLLM addresses these issues by introducing **a novel language modeling paradigm** that uses separators for segment infomation summarization, **optimizing from the pretraining to the inference**. Additionally, SepLLM naturally aligns with the hierarchical semantic structure of language (e.g., \",\" for segments, \".\" for sentences, \\n for paragraphs) and reduces KV usage during inference.\n\n> Q4. One issue in the ablation studies, the claim of better retrieval was introduced but no result was shown in the main body of the paper.\n\n**A4**.\nIndeed, we have already conducted **needle in a haystack** experiments and the results are shown in **Fig. 8-11** &  App. E, which demonstrate the retrieval ability of SepLLM. \n\n> Q5. Comparison to popular KVCache compression methods such as H20 (Neurips '23) and SnapKV (Arxiv'24)\n\n**A5**.\nPlease refer to Q3.\n\n> Q6. I saw a claim in the introduction that training-free methods are suboptimal but no evidence has been presented in the paper.\n\n**A6**. \nAs discussed in **Q3**, training-free methods are based on KV selection, an approximation to full attention, leading to\ninconsistency between training and inference phases.\n\n**Tab.13** can support our claim. Based on the well-trained Llama3-8B-Inst model, we fine-tuned for only 200 steps on the LongAlpaca dataset, and the results outperformed the original training-free SepLLM and even surpassed Vanilla.\n\n[1] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" ICLR 2024.  \n[2] Cai, Zefan, et al. \"Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.\" preprint arXiv:2406.02069 (2024).", "claims_and_evidence": "* \"From Figure 2,  it is seen that a given input context, seemingly “meaningless” separator tokens (such as commas, periods, exclamation marks, semicolons, etc.) that segment sequences receive higher attention scores compared to semantically meaningful tokens\n(such as nouns or verbs).\" This is a very interesting observation and is well used for the design.\n\n* \"Neighboring tokens usually help form locally smooth and coherent contexts, allowing the model to generate sentences that are reasonable within the immediate context.\" I like how the authors well defined all the different kinds of tokens.\n\n* Targeted masking for pre-training so that different kinds of tokens have different importance score follows well from the initial insights.", "methods_and_evaluation_criteria": "* Splitting the KV Cache into different blocks based on the kind of tokens seems like a promising idea and is well explained.\n* Evaluation is carried out on standard benchmark datasets and show very promising results\n* \"Our positional encoding strategy for streaming settings is the same as the state-of-the-art StreamingLLM\" more information on the exact technique would have been helpful, as the correct positional encoding seems to inspire the design a lot. Is the technique used the same as ROPE? Better?\n* Ablation studies show the importance of additional tokens, but it seems a better idea would have been to show the effect of separator tokens as that is the main claim for improvement.\n* How does SepLLM compare against well known KVCache compression techniques such as H20 and SnapKV?", "theoretical_claims": "Minor theoretical results are presented and the claims seem correct.", "experimental_designs_or_analyses": "* The experimental setup was well explained and evaluation on many parameters (quality, flops (compute) and size ) was carried out. \n* It was useful to see both the pre-training and post-training performance\n* One issue in the ablation studies, the claim of better retrieval was introduced but no result was shown in the main body of the paper.", "relation_to_broader_scientific_literature": "* The space of KV Cache compression is extremely interesting and has very promising implications on faster inference for LLMs. The scope and the methods in the paper are timely and well presented.", "essential_references_not_discussed": "* Comparison to popular KVCache compression methods such as H20 (Neurips '23) and SnapKV (Arxiv'24) has not been considered. Would love to see insights on that too.", "other_strengths_and_weaknesses": "* Interesting approach to see a KVCache compression technique which considers the KVCache being split into multiple sub KVCaches based on the properties of the tokens.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I'd encourage the authors to compare with well known KVCache compression techniques such  H20 (Neurips '23) and SnapKV (Arxiv'24) to better position the gains and have further clarity on how much the token-based KVCache partitioning helps. I saw a claim in the introduction that training-free methods are suboptimal but no evidence has been presented in the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I6YxBkgYzO", "reviewer": "Reviewer_kB2E", "summary": "This paper proposes SepLLM, an efficient Transformer-based architecture designed to accelerate inference in large language models (LLMs). The key insight is that separator tokens (such as punctuation and line breaks) disproportionately carry segment-level information, enabling the compression of other tokens without significant performance loss. The authors implement a sparse attention mechanism that retains only initial, separator, and neighboring tokens, substantially reducing the memory usage of the KV cache. Experimental results demonstrate that SepLLM achieves more than 50% reduction in KV cache size on GSM8K-CoT while maintaining competitive accuracy, and significantly improves inference speed and memory efficiency in both static and streaming tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n> Q1. The authors do not clearly explain or experimentally justify how these separator tokens were chosen.\n\n**A1**.\n We take the tokens in the 300B natural language dataset that exceed a certain word frequency threshold as separators, which are [“.”, “,”, “?”, “!”, “;”, “:”, “ ”, “\\t”, “\\n”].\n\n> Q2. The size of the neighboring tokens must be manually selected, requiring a trade-off between efficiency and performance.\n\n**A2**.\nIn our experiments, the size of neighboring tokens is set to be $m/8$ ~ $m/4$, where $m$ denotes the input sequence length. We find that the performance is still closed to that of full attention in these settings. While the size of neighboring tokens is set to be $m/2$ in StreamingLLM to maintain a similar performance.\n\n> Q3. The justification for the chosen set of separator tokens is not fully detailed, raising questions about the robustness of this selection across different tasks or languages.\n\n**A3**.\nIndeed, we use the same set of separators in all the experiments, including using lama-3-8B, Pythia-6.9B, Falcon-40B models and conducted on GSM8K, ARC, and MMLU datasets.\n\n> Q4. The experiments are largely limited to benchmarks like GSM8K-CoT and MMLU, leaving uncertainty about the method’s effectiveness on more complex, reasoning-intensive tasks.\n\n**A4**.\n\nAs suggested, we evaluate our SepLLM on MATH, which is a more complex benchmark, and the results are as follows.\nIt justifies the superiority of SepLLM in complex tasks.\n\n| | MATH | | | | | | | | |\n|---|---|---|---|---|---|---|---|---|---|\n| |**overall** | **r.KV(%)** |algebra | counting_and_prob | geometry |intermediate_algebra | num_theory | prealgebra | precalc | \n| Vanilla |27.32| 100.00|37.83| 24.26|18.58|12.62|21.11| 48.34|11.72|\n| SepLLM | 26.74 |32.4 | 37.99 | 24.05 | 19.83 |10.52|18.7|48.56|10.62 |\n| H$_2$O | 23.72 | 33.1| 34.71 | 17.72 | 15.03 | 11.41 | 19.63 | 41.79 | 8.24|\n| SnapKV | 25.68 | 33.1 | 37.49 | 21.94 | 16.91 | 10.41 | 20.37 | 45.58 |9.71|\n| PyramidKV | 25.1 | 33.1| 37.49|20.46| 16.49 | 9.97 |19.44 |44.55 |9.34|\n\n\n>   Q5. In the Related Work, after discussing the shortcomings of existing KV cache compression methods, The paper does not explicitly highlight how its approach overcomes these limitations.\n\n**A5**.\nAs discussed in Section 1 of our submitted paper, the existing KV cache compression methods are training-free approaches, which is inconsistent with training phase, resulting in approximation error.\n\n> Q6. How did you select the fixed set of separator tokens, and have you evaluated alternative sets?\n\n**A6**.\nFor the selection confusion, please refer to Q1. \nAs suggested, we evaluate our SepLLM with subsets of our previous setting on GSM8K-CoT, which are  [\",\" \".\"  \"?\"  \";\"] and [\".\"  \"?\"]. The experimental results are as follows. It shows that performance drops <0.5 when removing half of the separators. So SepLLM is robust against the separator selection. \n\n|                  | SepLLM (n=256)   | SepLLM (n=256)                | SepLLM (n=256) | StrmLLM (n=256) | StrmLLM (n=380) |\n|------------------|-------------------------------|----------------|-----------------|-----------------|-----------------|\n| flexible-extract | 77.18                         | 76.68          | 70.66           | 68.84           | 71.42 |\n| strict-match     | 77.18                         | 76.85          | 70.36           | 67.63           | 70.89 |\n| r.KV (%)         | 47.36                         | 37.92          | 36.44           | 31.92           | 47.54 |\n| separators       | \".\" \",\" \"?\" \"!\" \":\" \":\" \" \" \"\\t\" \"\\n\"  | \",\" \".\"  \"?\"  \";\" | \".\"  \"?\"          | - | - |\n\n> Q7. Can you provide insights into how the size of neighboring tokens (n) might be determined adaptively?\n\n**A7**.\nAs discussed in A2, we recommend the size of neighboring tokens to be $m/8$~$m/8$, where $m$ denotes the input sequence length.\n\nQ8. How does SepLLM perform on more challenging, reasoning-intensive tasks beyond GSM8K-CoT and MMLU?\n\n**A8**.\nPlease refer to Q4.", "claims_and_evidence": "The paper identifies separator tokens ([\".\", \",\", \"?\", \"!\", \";\", \":\", \" \", \"\\t\", \"\\n\"]) as critical for compressing segment-level information. \nHowever, the authors do not clearly explain or experimentally justify how these separator tokens were chosen.", "methods_and_evaluation_criteria": "The proposed methods, particularly the selective retention of initial, separator, and neighboring tokens, are well-aligned with the goal of reducing computational complexity in LLMs. \nAdditionally, employing well-established benchmarks such as GSM8K-CoT, MMLU, and PG19 provides a clear and appropriate evaluation framework for the model's performance.", "theoretical_claims": "I reviewed the theoretical proofs provided, particularly the universal approximation analysis in Appendix G.", "experimental_designs_or_analyses": "The experimental design, particularly using GSM8K-CoT and MMLU, is well-structured to demonstrate efficiency and reasoning improvements. However, it remains unclear whether the approach can maintain performance on more challenging tasks requiring extended reasoning (e.g., Math 500), and further experiments in this area would be beneficial.", "relation_to_broader_scientific_literature": "The paper builds on prior research in KV cache compression and sparse attention—such as FastGen, SnapKV, and StreamingLLM—by introducing a novel approach that leverages separator tokens to compress segment-level information. This contribution extends methods like BigBird and other sparse attention mechanisms by showing that dynamic, data-dependent token selection can significantly reduce computational overhead while maintaining performance.", "essential_references_not_discussed": "To the best of my knowledge, the paper has cited all essential references necessary to understand the context and key contributions of the work.", "other_strengths_and_weaknesses": "Strengths:\n1.The paper introduces an innovative approach by leveraging separator tokens to compress segment-level information, which is a novel idea in efficient LLM design.\n2.The method is thoroughly evaluated across various settings (training-free, training-from-scratch, post-training, and streaming), demonstrating its practical effectiveness.\n3.The approach achieves substantial computational savings—significantly reducing KV cache usage—while maintaining competitive performance.\n\nWeaknesses:\n1.The size of the neighboring tokens must be manually selected, requiring a trade-off between efficiency and performance.\n2.The justification for the chosen set of separator tokens is not fully detailed, raising questions about the robustness of this selection across different tasks or languages.\n3.The experiments are largely limited to benchmarks like GSM8K-CoT and MMLU, leaving uncertainty about the method’s effectiveness on more complex, reasoning-intensive tasks.", "other_comments_or_suggestions": "In the Related Work, after discussing the shortcomings of existing KV cache compression methods, the paper does not explicitly highlight how its approach overcomes these limitations.", "questions_for_authors": "1.How did you select the fixed set of separator tokens, and have you evaluated alternative sets? \n\n2.Can you provide insights into how the size of neighboring tokens (n) might be determined adaptively?\n\n3.How does SepLLM perform on more challenging, reasoning-intensive tasks beyond GSM8K-CoT and MMLU?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel attention mechanism that allows LLMs to attend to only a subset of previous tokens (a few initial tokens, separator tokens, and local neighboring tokens). This approach improves both training and inference speed, and reduces the size of the KV cache necessary at inference time. Reviewers agree the paper is well motivated, the method is well explained, and that the evaluations are thorough and show the proposed approach is strong. The initial reviews raised valid concerns, including comparisons to additional baselines (like uniformly subsampling tokens to attend to), and robustness of the design choices (e.g. the separator tokens). The authors have addressed most of these in the rebuttal, and reviewers have appropriately raised their scores. Given the performance of the method and that reviewers agree on acceptance, we recommend this paper for acceptance at ICML.
Decision: Accept (poster)