Efficient Streaming Language Models with Attention Sinks
Authors: Guangxuan Xiao
Yuandong Tian
Beidi Chen
Song Han
Mike Lewis
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Length Extrapolation
Efficiency
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We enable LLMs to work on infinite-length texts without compromising efficiency and performance.

Abstract: Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges.
Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory.
Secondly, popular LLMs cannot generalize to longer texts than the training sequence length.
Window attention, where only the most recent KVs are cached, is a natural approach --- but we show that it fails when the text length surpasses the cache size.
We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a ``sink'' even if they are not semantically important.
Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning.
We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2$\times$ speedup.
Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Published as a conference paper at ICLR 2024
EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS
Guangxuan Xiao
Yuandong Tian? Beidi Chen?
Song Han
Mike Lewis?
Massachusetts Institute of Technology Meta AI 3 Carnegie Mellon University NVIDIA
https://githab. momm han lab/ st st 11m
ABSTRACT
Deploying Large Language Models (LLMs) in streaming applications such multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens "sink" even they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize infinite sequence length without any fine-tuming, We show that Streaming can enable 1lama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placcholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided in the link
but
INTRODUCTION
Large Language Models (LLMs) Radford et al., 2018; Brown et al. 2020: Zhang et al. 2022
OpenAl 2023 Touvron al.) 2023a b) becoming ubiquitous, powering many natural language
processing applications such as dialog systems (Schulman et al. 2022 Taori et al. 2023 Chiang et al. 2023). document summarization (Goyal & Durrett 2020 Zhang al. 2023a). code completion (Chen
ct al. 2021; Rozière et al.] (023) and question answering (Kamallo et al. (023). To unleash the full potential of pretrained LLMs, they should be able efficiently and accurately perform long sequence generation. For example, an ideal ChatBot assistant can stably work over the content of recent day-long conversations. However, it is very challenging for LLM generalize longer sequence lengths than they have been pretrained on, c.g., 4K for Llama-2 Touvron et al. (0023b).
The reason that LLMs are constrained by the attention window during pre-training. Despite substantial efforts to expand this window size (Chen et al. [2033] kaiokendev 2023 Peng et al. 2023
and improve training (Dao et al.] 2202;) Dao 2023) and inference (Pope al., 2022
Xiao et al. 2023
Anagnostidis al.) 2023 Wang et al. 2021 Zhang et al. (023b) efficiency for lengthy inputs, the acceptable sequence length remains intrinsically finite, which doesn't 't allow persistent deployments.
In this paper, we first introduce the concept of L.LM streaming applications and ask the question:
Can we deploy an LLM for infinite-enngth inputs without sacrificing efficiency and performance?
Part of the work done during an internship Meta AI.

---

Published as a conference paper at ICLR 2024
(c) Sliding Window w/ Re-computation
(a) Dense Attention
(b) Window Attention
(d) StreamingLLM (ours)
Cuncal Token
Attemiion
Innosted
Trached iekees
cachal
loketa
tekem
teken
O(T3)x PPL: 5641x
O(TL) PPL: 5158X
O PPL: v
O(TL) V PPL: 5,40 V
poor efficiency and perfomannce on long tost.
Bresks when initial
to ro-comyure cache for each incoming token.
Can perform efficient and stable language modeling on long texts.
tokens are ovieted.
Illustration of StreamingI.LM vs. existing methods. The language model, pre-trained on texts of length L. predicts the Tth token (T' > L). (a) Dense Attention has (TT)) time complexity and an increasing cache size. Its performance decreases when the text length exceeds the pre-training text length. (b) Window Attention eaches the most recent tokens' KV. While efficient inference, performance declines sharply once the starting tokens' keys and values are evicted. (c) Sliding Window with Re-computation rebuilds the KV states from the L recent tokens for cach new token. While it performs well on long texts, its OTT)²) complexity, stemming from quadratic attention in context re computtaiion, makes it considerably slow. (d) StreamingLLM keeps the attention sink (several initial tokens) It's efficient and offers stable performance on extended texts. Perplexities are measured using the Llama-2-13B model on the first book (65K tokens) in the PG-19 test set.
Figure I:
stable attention computation, combined with the recent tokens.
When applying LLMs for infinite input streams, two primary challenges arise:
1. During the decoding stage, Transformer-basee LLMs cache the Key and Value states (KV) all previous tokens, as illustrated in Figure I (a), which can lead to excessive memory usage and increasing decoding latency (Pope et al. 2022).
2. Existing models have limited length extrapolation abilities, i.c., their performance de- grades (Press al. 2022 Chen et al. (202)) when the sequence length goes beyond the attention window size set during pre-training
An intuitive approach, known as window attention (Beltagy et al. (020) (Figure maintains only fixed-size sliding window on the KV states of most recent tokens. Although it ensures constant memory usage and decoding speed after the cache is initially filled, the model collapses once the sequence length exceeds the cache size, i.e., even just evicting the KV of the first token, as illustrated in Figure 3 Another strategy is the sliding window with re-computation (shown in Figure Ic), which rebuilds the KV states of recent tokens for each generated token. While offers strong performance, this approach is significantly slower due to the computation of quadratic attention within its window, making this method impractical for real-world streaming applications.
To understand the failure of window attention, we find interesting phenomenon of autoregressive LLMs: a surprisingly large amount of attention score is allocated to the initial tokens, irrespective of their relevance to the language modeling task, as visualized in Figure 2] We term these tokens "attention sinks". Despite their lack of semantic significance, they collect sigmificant attention scores. We attribute the reason the Softmax operation, which requires attention scores to sum up one for all contextual tokens. Thus, even when the current query does not have a strong match in many previous tokens, the model still needs to allocate these unneeded attention values somewhere so sums to one. The reason behind initial tokens as sink tokens is intuitive: initial tokens are visible to almost all subsequent tokens because of the autoregressive language modeling nature, making them more readily trained to serve as attention sinks.
Based on the above insights, we propose StreamingLLM, a simple and efficient framework that enables LLMs trained with a finite attention window to work on text of infinite length without fine- tuning. StreamingL_LMM exploits the fact that attention sinks have high attention values, and preserving them can maintain the attention score distribution close to normal. Therefore, StreamingLLLM simply keeps the attention sink tokens' KV (with just 4 initial tokens sufficing) together with the sliding window's 's KV to anchor the attention computation and stabilize the model's performance. With StreamingLLMM, models including Llama-2-f7 2-17, 13, 70JB, MPT-I7, 30JB, Falcon-[7, 40JB, and Pythia- [2.9,6.9, 12 can reliably model 4 million tokens, and potentially even more. Compared with the only viable baseline, sliding window with recomputation, Streaming achieves up to 22.2x x speedup, realizing the streaming use of L.MMs.
2

---

Published as a conference paper at ICLR 2024
Layer Layer Head
Loyel
Figure 2: Visualization of the average attention logits in Llama-2-7B over 256 sentences, each with length of 16. Observations include: (1) The attention maps the first two layers (layers 0 and 1) exhibit the "local" pattern, with recent tokens receiving more attention. (2) Beyond the bottom two layers, the model heavily attends to the initial token across all layers and heads.
Furthermore, we confirm our attention sink hypothesis and demonstrate that language models can be pre-trained to require only single attention sink token for streaming deployment. Specifically, we suggest that an extra learnable token at the beginning of all training samples can serve designated attention sink. By pre-training 160-million parameter language models from scratch, we demonstrate that adding this single sink token preserves the model's performance in streaming cases. This stands in contrast to vanilla models, which necessitate the reintroduction of multiple initial tokens as attention sinks to achieve the same performance level.
Finally, we emphasize that Streamingl LLM efficiently generates coherent text from tokens within the KV cache without extending the LLMs' context length. It suits continuous operation needs with minimal memory use and past data reliance. Additionally, StreamingLLM can complement context extension methods to increase the attendable recent context.
2 RELATED WORK
Extensive research has been done applying LLMs to lengthy texts, with three main areas of focus: Length Extrapolation, Context Window Extension, and Improving LLMs Utilization of Long Text. While seemingly related, it's worth noting that progress in one direction doesn't 't necessarily lead to progress in the other. For example, extending the context size of LLMs doesn't 't improve the model's performance beyond the context size, and neither approach ensures effective use of the long context. Our StreamingLLM framework primarily lies in the first category. where LLMs are applied to text significantly exceeding the pre-training window size, potentially even of infinite length. We do not expand the attention window size of LLMs or enhance the model's 's memory and usage on long texts. The last two categories are orthogonal to our focus and could be integrated with our techniques.
Length extrapolation aims to enable language models trained on shorter texts to handle longer ones during testing. A predominant avenue of research targets the development of relative position encoding methods for Transformer models, enabling them to function beyond their training window. One such initiative is Rotary Position Embeddings (RoPE) (Su et al. 202T). which transforms the queries and keys in every attention layer for relative position integration. Despite its promise, subsequent research (Press et al., 2022 Chen et al., (023) indicated its underperformance on text that exceeds the training window. Another approach, ALiBi (Press et al., (022), biases the query-key attention scores based on their distance, thereby introducing relative positional information. While this exhibited improved extrapolation, our tests on MPT models highlighted a breakdown when the text length was vastly greater than the training length. Current methodologies, however, have yet to achieve infinite length extrapolation, causing existing LLMS to fit for streaming applications.
Context Window Extension centers on expanding the LLMs' context window, enabling the process- ing of more tokens one forward pass. A primary line of work addresses the training efficiency problem. Given the attention to computation's 's quadratic complexity during training, developing a long-context LLM is both computational and memory challenge. Solutions have ranged from system-focused optimizations like FlashAttention (Dao et al., 2022 Dao 2023), which accelerates attention computation and reduces memory footprint, to approximate attention methods (Zaheer al. 2020b: Beltagy et al. 2020 Wang et al. 2020 Kitaev et al. 2020) that trade model quality for efficiency. Recently, there has been a surge of work on extending pre-trained LLMs with RoPE (Chen et al. 2023 kaiokendev 2023 bloc97 2023 Peng et al. 2023), involving position interpolation and fine-tuning. However, all the aforementioned techniques only extend LLMs' context window to a limited extent, which falls short of our paper's primary concern of handling limitless inputs.
3

---

Published as a conference paper at ICLR 2024
Dense Attontion
Window Attontion
w/ Re-computation
Sliding Window
StroamingL
Liama-277
Pythia-128
Ri
Q Cande Aan
Cocre Size
Largih
06
108
20c
126
20k 00
54
106
35K
10c
Figure 3: Language modeling perplexity on texts with 20K tokens across various LLM. Observations reveal consistent trends: (1) Dense attention fails once the input length surpasses the pre-training attention window size. (2) Window attention collapses once the input length exceeds the cache size, i.e., the initial tokens are evicted. (3) StreamingI demonstrates stable performance, with its perplexity nearly matching that of the sliding window with re-computaiion baseline.
Improving LLMs Utilization of Long Text optimizes LLMs to better capture and employ the content within the context rather than merely taking them inputs. As highlighted by Liu er al. and et al.] success in the previously mentioned two directions does not necessarily translate to competent utilization of lengthy contexts. Addressing this effective usage of prolonged contexts within LLMs is still a challenge. Our work concentrates on stably harnessing the most recent tokens, enabling the seamless streaming application of LLMs.
3
STREAMINGLLM
3.1 THE FAILURE OF WINDOW ATTENTION AND ATTENTION SINKS
While the window attention technique offers efficiency during inference, it results in an exceedingly high language modeling perplexity. Consequently, the model's performance is unsuitable for deploy- ment in streaming applications. In this section, we use the concept of attention sink to explain the failure of window attention, serving as the inspiration behind StreamingL..M.
Identifying the Point of Perplexity Surge. Figure shows the perplexity of language modeling on a 20K token text. It is evident that perplexity spikes when the text length surpasses the cache size, led by the exclusion of initial tokens. This suggests that the initial tokens, regardless of their distance from the predicted tokens, are crucial for maintaining the stability of LLMs.
Why do LLMs break when removing initial tokens' KV? We visualize attention maps from all layers and heads of the Llama-2-7B and models in Figure We find that, beyond the bottom two layers, the model consistently focuses the initial tokens across all layers and heads. The implication is clear: removing these initial tokens' KV will remove a considerable portion of the denominator in the SoftMax function (Equation [ in attention computation. This alteration leads to significant shift in the distribution of attention scores away from what would be expected in normal inference settings.
SoftMax(z): = Ø 2
> T..J ......, N
There are two possible explanations for the importance of the initial tokens in language modeling: (1) Either their semantics are crucial, or (2) the model learns a bias towards their absolute position. To distinguish between these possibilities, we conduct experiments (Table wherein the first four tokens are substituted with the linebreak token "In". The observations indicate that the model still significantly emphasizes these initial linebreak tokens. Furthermore, reintroducing them restores the language modeling perplexity to levels comparable to having the original initial tokens. This suggests that the absolute position of the starting tokens, rather than their semantic value, holds greater significance.
LLMs attend Initial Tokens as Attention Sinks. To explain why the model disproportionately focuses on initial the concept of "attention sink". The nature of the SoftMax function (Equation] ! prevents all attended tokens from having zero values. This requires aggregating some information from other tokens across all heads in all layers, even if the current embedding has sufficient self contained information for its prediction. Consequently, the model tends to dump unnecessary attention values to specific tokens. A similar observation has been made in the realm of quantization outliers (Xiao et al., 2023 Bondarenko et al. 2023), leading the proposal of SoftMaxxOOff-y-OOn (Miller 2023 potential remedy.
of their semantic relevance to language modeling, we introduce
4

---

Published as a conference paper at ICLR 2024
Window attention has poor per- formance on long text. The perplexity is restored when we reintroduce the initial four tokens alongside the recent 1020 kens (4+1020). Substituting the original four initial tokens with linebreak tokens "In" (4"'n"+0200 achieves comparable perplexity restoration. Cache config x+y denotes adding x initial tokens with y recent tokens. Perplex- itics are measured on the first book (65K to- kens) in the PG19 test set.
Table 1:
Table 2: Effects of reintroduced initial token numbers on StreamingLLM. (1) Window attention (0+y) has a drastic in- crease in perplexity. (2) Introducing
two initial tokens doesn't fully restore model perplexity, showing that the model doesn't 't solely use the first token as the attention sink. (3) Intro- ducing four initial tokens generally suffices; f'urther additions have diminishing returns. Cache config x+y denotes adding initial tokens y recent tokens. Perplexities are evaluated on 400K tokens in the concatenated PG19 test set.
Cache Config 0+2048
1+2047
2+2046 4+2044
8+2040
Falcon-7B
17.90
12.12
12.12
12.12
12.12
Llama-2-13B
PPL. (1)
MPT-7B
460.29
14.99
15.00
14.99
14.98
+ 1024 (Window) 5158.07
Pythia-120 12B
21.62
11.95
12.09
12.09
12.02
4 + 1020
5.40
4"2n"+0020
5.60
Cache Config
0+4096
1 1+4095
2+4094
4+-4092
8+4088
Llama-2-7B 3359.95
11.88
10.51
9.59
9.54
Why do various autoregressive L.M,s, such as Llama-2, MPT, Falcon, and Pythia, consistently focus on initial tokens as their attention sinks, rather than other tokens? Our explanation is straightforward: Due to the sequential nature of autoregressive language modeling, initial tokens are visible to all subsequent tokens, while later tokens are only visible to limited set of subsequent tokens. As result, initial tokens are more easily trained to serve as attention sinks, capturing unnecessary attention.
We've ve noted that LLMs are typically trained to utilize multiple initial tokens attention sinks rather than just one. As illustrated in Figure - the introduction of four initial tokens, as attention sinks, suffices restore the LLM's performance. In contrast, adding just one or two doesn't 't achieve full recovery. We believe this pattern emerges because these models didn't tt include a consistent starting token across all input samples during pre-training. Although Llama-2 does prefix each paragraph with a "
" token, it's applied before text chunking, resulting in mostly random token occupying the zeroth position. This lack of uniform starting token leads the model to use several initial tokens as attention sinks. We hypothesize that by incorporating a stable learnable token at the start of all training samples, it could singularly act as a committed attention sink, eliminating the need for multiple initial tokens to ensure consistent streaming. We will validate this hypothesis in Section 3.3
3.2 ROLLING KV CACHE WITH ATTENTION SINKS
Genaseing Takes
2
3
4
5
6
7
To enable LLM streaming in already trained LLMs, we propose straightforward method that can recover win- dow attention's 's perplexity without any model finetuning. Alongside the current sliding window tokens, we reintro- duce a few starting tokens' KV in the attention computa- Figure 4: The KV cache of StreamingL tion. The KV cache in Streaming LLM can be conceptually divided into two parts, as illustrated in Figure 4 (1) Attention sinks (four initial tokens) stabilize the attention computation; 2) Rolling KV Cache retains the most recent tokens, crucial for language modeling. StreamingLLMM design is versatile and can be seamlessly incorporated into any autoregressive language model that employs relative positional encoding, such as RoPE (Su et al. 2021) and ALiBi (Press et al., 2022).
Genertina
2
3
4
5
6
7
8
Geoeroting Taken
2
3
7
8
Attention Sinks
Exceted takens Rolling KV Coohe
When determining the relative distance and adding positional information to tokens, StreamingLLM, focuses on positions within the cache rather than those in the original text. This distinction is crucial for Streamingl LLM's performance. For instance, if the current cache (Figure 4 has tokens [0, 1, 2, 3, 6, 7, 8) and is in the process of decoding the 9th token, the positions assigned are [0, 1, 2, 3, 4, 5, 6, 71, rather than the positions in the original text, which would be [0. 1, 3, 6, 8, 9].
For encoding like RoPE, we cache the Keys of tokens prior to introducing the rotary transformation. Then, we apply position transformation to the keys in the rolling cache at each decoding phase. On the other hand, integrating with ALiBi is more direct. Here, the contiguous linear bias is applied instead of 'jumping' bias to the attention scores. This method of assigning positional embedding within the cache is crucial to StreamingLLM'' ' functionality, ensuring that the model operates efficiently even beyond its pre-training attention window size.
5

---

Published as a conference paper at ICLR 2024
3.3 TRAINNGNN LLMs WITH ATTENTION SINKS
Table 3: Comparison of vanilla attention with prepend- ing zero token and learnable sink token during pre- training. To ensure stable streaming perplexity, the vanilla model requires several initial tokens. While Zero Sink shows a slight improvement, it still needs other ini- tial tokens. Conversely, the model trained with learn- able Sink Token shows stable streaming perplexity with only the sink token added. Cache config a+y denotes adding x initial tokens with y recent tokens, Perplexity is evaluated on the first sample in the PG19 test set.
As elaborated in Section 3.1 significant rea- son for the model's excessive attention to multi- ple initial tokens is the absence of a designated sink token to offlood excessive attention scores. Due to this, the model inadvertently uses glob- ally visible tokens, primarily the initial ones, as attention sinks. A potential remedy can be the intentional inclusion of global trainable attention sink token, denoted as a "Sink Token", which would serve as a repository for unneces- sary attention scores. Alternatively, replacing the conventional SoftMax function with vari- ant like SoftMax--off-y-One (Miller 2023).
Cache Contig
0+1024
1+1023
2+1022
4+1020
Vanilla
27.87
18.49
18.05
18.05
Zero Sink
29214
19.90
18.27
18.01
Learnable Sink
1235
18.01
18.01
18.02
SoftMax; (z), 1
Σド= e"
which does not require the attention scores on all contextual tokens to sum up to one, may also be effective. Note that SoftMax; is equivalent to prepending token with an all-zero Key and Value features in the attention computation. We denote this method as "Zero Sink" to fit our framework.
For validation, we pre-train three language models with 160 million parameters from scratch under identical settings. The first model utilizes the standard SoftMax attention (Vanilla), the second replaced the regular attention mechanism with SoftMax, (Zero Sink), and one prepending learnable placeholder token (Sink Token) in all training samples. As shown in Table 3 while the zero sink alleviates the attention sink problem to some extent, the model still relies on other initial tokens as attention sinks. Introducing a sink token is highly effective in stabilizing the attention mechanism. Simply pairing this sink token with recent tokens sufficiently anchors the model's performance, and the resulting evaluation perplexity is even marginally improved. Given these findings, we recommend training future LLMs with a sink token in all samples to optimize streaming deployment.
4 EXPERIMENTS
We evaluate Streaming using four prominent recent model families: Llama-2 Touvron et al. 2023b), MPT (Team, 2023), PyThia (Biderman et al.) 2023). and Falcon (Almazrouei et al. 2023). Notably, Llama-2, Falcon, and Pythia incorporate RoPE (Su et al., (202). whereas MPT employs ALiBi (Press et al. 2022) two of the most influential position encoding techniques in recent research. Our diverse model selection ensures the validity and robustness of our findings. We bench- mark StreamingLLM against established baselines such as dense attention, window attention, and the sliding window approach with re-computation. In all subsequent experiments with StreamingLLMM, we default to using four initial tokens as attention sinks unless stated otherwise.
4.1 LANGUAGE MODELING ON LONG TEXTS ACROSS LLM FAMILIES AND SCALES
We firstly evaluate StreamingLLMM' language modeling perplexity using the concatenated PG19 (Rae) al., (220)) test set, which contains 100 long books. For Llama-2 models, the cache size is set at 2048, while for Falcon, Pythia, and MPT models, it's set at 1024. This is half the pre-training window size chosen to enhance visualization clarity.
Figure 3 illustrates that StreamingI.LMM can match the oracle baseline (sliding window with re- computation) in terms of perplexity on texts spanning 20K tokens. Meanwhile, the dense attention technique fails when the input length exceeds its pre-training window, and the window attention technique struggles when the input length surpasses the cache size, leading to the eviction of the initial tokens. In Figure we further substantiate that StreamingLLM can reliably handle exceptionally extended texts, encompassing more than 4 million tokens, across a spectrum of model families and scales. This includes Llama-2-4 2-1, falcon-[7,4001: (yythia[[28866999 12JB, and MPT-[7,30JBB

---

Published as a conference paper at ICLR 2024
Liama-2 (Streaming.LUM
Pytnia Streamnng
Racoon (SreeminngL
(SheaminggLMM
1y0ta 2.00
Pyttia-d.00
Pytti
66
244 4M CM
15
204
344 3M
30
24
3M
Figure 5: Language modeling perplexity of StreamingLLM on super long texts with 4 million tokens across various LLM families and scales. The perplexity remains stable throughout. We use the concatenated test set of PG19 (100 books) to perform language modeling, with perplexity fluctuations due to book transitions.
Lac
Lager ttad
Pre-Trained without Sink Token
Pre-Truined with Sink Token
Figure 7: Visualization of average attention logits over 256 sentences, each 16 tokens long, comparing models pre- pre-trained without (left) and with (right) a sink token. Both maps show the same layers and heads. Key observations: (1) Without a sink token, models show local attention in lower layers and increased attention to initial tokens in deeper layers. (2) With sink token, there is clear attention directed at it across all layers, effectively collecting redundant attention. (3) With the presence of the sink token, less attention is given other initial tokens, supporting the benefit of designating the sink token to enhance the streaming performance.
4.2 RESULTS OF TRA-TIINNIN WITH SINK TOKEN
To validate our suggestion that introducing a sink token to all pre-training samples improves stream- ing LLMs. we trained two language models, each with 160 million parameters, under identical conditions. While one model adhered to the original training settings, the other incorporated sink token at the start of every training sample. Our experiments employed the Pythia-160M, (Bider) man et al., 2023) codebase and followed its training recipe. We train the models
8xA6000 NVIDIA GPU server using the deduplicated Pile (Gao et al., 2020) dataset. Apart from reducing the training batch size to 256, we retained all Pythia training configurations, including learning rate schedules, model initialization, and dataset permutations. Both models were trained for 143,000 steps.
2.8
Table 4: Zero-shot accuracy (in %) across 7 NLP benchmarks, including ARC-[Challenge, Easy], HellaSwag, LAMBADA, OpenbookQA, PIQA, and Winogrande. The inclusion of sink token during pre-training doesn'i t harm model performance.
Vanills
lns
2.7
Sink Taken
Methods
ARC-c
ARC-e
HS
LBD
OBQA
PIQA
WG
0
20
40
50
80
100
120
140
K.ttps
Figure
Pre-training loss
Vanilla
18.6
45.2
29.4
39.6
16.0
62.2
50.1
curves of models w/ and w/o sink tokens. Two models have simi- lar convergence trend.
+Sink Token
19.6
45.6
29.8
39.9
16.6
62.6
50.8
Convergence and Normal Model Performance. Including sink token during pre-training has no negative impact on model convergence and subsequent performance on a range of NL.P benchmarks. As depicted in Figure 6. models trained with sink token exhibit similar convergence dynamics compared to their vanilla counterparts. We evaluate the two models on seven diverse NLP bench- marks, including ARC-IChallenge, Easy] (Clark al., 2018), HellaSwag (Zellers et al. 2019). LAMBADA (Paperno et al., 2016), OpenbookQA (Mihaylov et al. 2018), PIQA (Bisk et al. 2020 and Winogrande (Sakaguchi et al.) 2019). As shown Table 4 the model pre-trained with sink token performs similarly to that trained using the vanilla approach.
Streaming Performance. As illustrated in Table the streaming perplexities differ between models trained using traditional methods and those augmented with sink token. Remarkably, the vanilla model requires the addition of multiple tokens as attention sinks to maintain stable streaming perplexity. contrast, the model trained with sink token achieves satisfactory streaming performance using just the sink token.
7

---

Published as a conference paper at ICLR 2024
Table Accuracy (in %) on the ARC-[Easy, Challenge] datasets. Questions were concatenated and answered in a streaming manner to mimic a real-world chat setting. The dense baseline fails due to Out-of -ft-ooy (OOM) errors. Window attention has poor accuracy. StreamingLLM has comparable results with the one-shot sample-by-sampl baseline. Window attention and Streamingl . use cache sizes of 1024.
Model
Llama-2-7B- 2--a--a 1-lmm-2-113--0-
(lama-2-770-CCie
Dataset
Arc-E
Arc-C
Arc-E
Arc-C
Arc-E
Are-C
One-shot
71.25
53.16
78.16
63.31
91.29
78.50
Dense
OOM
Window
3,58
1.39
0.25
0.34
0.12
0.32
StreamingLLM
71.34
55.03
80.89
65.61
91.37
80.20
Dense Attention
Window Attention
StreamingLLM
14-
Ubmo- Thh
1.0
Ubme-2-13b-Char
1.0
LongChat-7b 5:322
1.0
0.0
u.
us
Eax
00
25K
408
03.
BOK
120K
120K
0.0
20K
40X
03
BOC
120K
120K
0.0
20C
40C
20K
20K
10CK
120K
0.9
30K
400
206
20K
100K 120K
Figure 9: Performance on the StreamEval benchmark. Accuracies are averaged 100 samples.
Input Content
Attention Visualization. Figure 7 contrasts attention maps for models pre-trained with and without a sink token. The model without the sink token, similar to Llama-2-7B (Figure[), shows early-layer local attention and deeper-layer focus on initial tokens. In contrast, models trained with a sink token consistently concentrate on the sink across layers and heads, indicating an effective attention offloading mech- anism. This strong focus on the sink, with reduced attention to other initial tokens, explains the sink token's 's efficacy in enhancing model's 's streaming performance.
Belox is a record of Tines want you to rwanster. The REGISTER__ONTTR lonstting 9 Lsnes.)
in Line i is <b88ss
The REGISTER CONTINT Line 10 is <248780
fonitting 8 Lines_.)
The REG1S Ten, CONTENT Line 26 is <456835
Suery: The REGISTER, CONTENT in Linw iN The REGISTER, CONTINT Line <291850
tonisting Lines_.
The REGISTER, CERTCNT Line +1658-
RESISTER
The SER_STEECCNCETTTT Line [onteting
The REGISTER CONTENT
<24579-
Suery: RESISTER, CONTENT
20 18
[onitting rewaiiing 5467 Yinss..
Desired Ourgut
['29066-", " "248700" "x45683>"" -11
Figure 8: The first sample in StreamEval.
4.3 RESULTS ON STREAMING QUESTION ANSWERING WITH INSTRUCTION-TUNDE MODELS
To show StreamingLLM's real-world applicability, we emulate multi-round question-ansseringg using instruction-tume -uned LLMs, commonly used in real-world scenarios.
We first concatenate all question-answer pairs from the ARC-[Challend Easy] datasets, feed the continuous stream to 21..ma2-(7,13,70088 -Chtt models, and assess model completions at each answer position using an exact match criterion. As table mable5iiniionn indicates, dense attention results in Out-of- (OOM) errors, showing it unsuitable for this setting. While the window attention method works efficiently, it exhibits low accuracy due to random outputs when the input length exceeds the cache size. Conversely, StreamingLLM excels by efficiently handling the streaming format, aligning with the one-shot, sample-by-sample baseline accuracy.
Highlighting a more fitting scenario for StreamingL.LMM we introduce a dataset, StreamEval, inspired by the LongEval (Li et al.) 2023) benchmark. As depicted in Figure diverging from LongEval's single query over a long-span setup. we query the model every 10 lines of new information. Each query's 's answer is consistently 20 lines prior, reflecting real-world instances where questions typically pertain to recent information. As illustrated in Figure 9 L.MMs employing StreamingL maintain reasonable accuracy even as input lengths approach 120K tokens. In contrast, both dense and window attention fail at the pre-training text length and the KV cache size, respectively. Additionally, we utilize two context- extendedd models, LongChat-7brr 7 (Li al. 2023) and Llama-2-7B-32K- Instruct (Together 2023), to show that StreamingLLM can complement context extension techniques. Within EtreaminLLMM. context extension means broadening the maximum cache size of streaming LLMss, enabling the capture of broader local information.
8

---

Published as a conference paper at ICLR 2024
Sliding Window with Re-computation
StreamingL.MM
Latency (its)
1200
1600
1411
Menory (GB))
14
1333
1414
2250
2355
Menory (GBB)
25-25
25.26 2622
800
1500
140
67:1
10211
99.8
169k
361
256
$12
1976
70488
4006
246
$12
1874
7063
4036
216
512
1004
7048
4006
216
517
1854
7046
4096
Llama-2-7B
Llama-2-15
2-13B
Figure 10: Comparison of per-token decoding latency and memory usage between the sliding window approach with re-computation baseline and StreamingL.MM, plotted against the cache size (attention window size) on the X-axis. StreamingLL.M delivers remarkable speedup of up 22.2 per token and retains a memory footprint similar to the re-computation baseline.
4.4 ABLATION STUDIES
Numbers of Initial Tokens. In 2 we ablate the effect of adding varying numbers of initial tokens with recent tokens on the streaming perplexity. The results show the insufficiency of introduc- ing merely two initial tokens, whereas threshold of four initial tokens appears enough, with subsequent additions contributing marginal effects. This result justifies our choice of introducing initial tokens attention sinks in StreamingLLLM.
Cache Sizes. In Table we evaluate cache size's impact on StreamingLLM'ss perplex- ity. Contrary to intuition, increasing the cache size doesn't consistently lower the lan- guage modeling perplexity. This inconsistency shows a potential limitation where these mod- els might not maximize the utility of the entire context they receive. Future research ef- forts should target enhancing these models' capabilities to utilize extensive contexts better.
Table Effects cache size on Streaming performance. Increasing the cache size in StreamingLLM doesn't consistently yield de- crease in perplexity, showing these models may fully utilize the provided context. Cache config x+y denotes adding initial tokens with y recent tokens. Perplexity is evaluated on 400K tokens in the concatenated PG19 test set.
4.5 EFFICENCY RESULTS
We benchmark StreamingLLM'ss decoding latency and memory usage against the sliding window with re-computation, which is the only baseline with ac- ceptable quality. Both methods are implemented using the Huggingface Transformers library (Wolf cct al., 2020) and tested on a single NVIDIA A6000 GPU using the Llama-2-78 2- and Llama-2-133 models. As shown Figure 10 as the cache size increases, StreamingLLLM' decoding speed has a linear growth. The sliding window with re-computation baseline has a quadratic rise in decoding latency. Thus, StreamingLLM achieves an impressive speedup, reaching up to 22.2x x per token. Despite its reduced latency, Streaming sustains memory footprint consistent with the re-computation baseline.
Cache
4+252
4+508
4+1020
4+2044
Falcon-7B
13.61
12.84
12.34
12.84
MPT-7B
14.12
14.25
14.33
14.99
Pythia-12B
13.17
12.52
12.08
12.09
Cache
4+508
4+1020
4+2044
4+4092
Alama-2-78
9.73
9.32
9.08
9.59
5 CONCLUSION
Deploying LLMs in streaming applications is urgently needed but comes with challenges due to efficiency limitations and reduced performance with longer texts. Window attention provides a partial solution, but its performance plummets when initial tokens are excluded. Recognizing the role of these tokens as "attention sinks", we introduced StreamingLLM -a simple and efficient framework that enables LLMs to handle unlimited texts without fine-tuning. By adding attention sinks with recent tokens, StreamingLLM can efficiently model texts of up to million tokens. We further show that pre-training models with dedicated sink token improve the streaming performance. StreamingLL.M firstly decouples the LLM's pre-training window size and its actual text generation length, paving the way for the streaming deployment of L.MMs.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
All findings presented in this paper can be reproduced. We have made our code and datasets available in this github repo, The models used in this paper are all openly available, and provide references to access them. Details regarding our experiments, including hyperparameters, training protocols, and evaluation methods, can be found in the Experiments section (Section 4.. We are confident that with the provided resources, readers can reproduce the entirety of our presented results.
IMPACT STATEMENT
Streaming LLM LLM has been widely adopted by various LLM serving solutions including NVIDIA TensorRT-LLA Intel Extension for Transformers, HuggingFace Transformers MLC LLM etc.
ACKNOWLEDGEMENTS
This work is supported by MIT-IBM Watson AI Lab, Amazon and MIT Science Hub, National Science Foundation. We thank Angela Li for writing suggestions and demo making, Jingwei Zuo for proofreading, and Xiuyu Li for the suggestion on notations.
REFERENCES
Joshua Ainslic, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Ete: Encoding long and structured inputs in transformers, 2020.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with otate-of-the-art performance. 2023.
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14500, 2023.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. arXiv:2004.05150.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourts AAAI Conference on Artificial Intelligence, 2020.
bloc97. AwK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023. URL https:/wwww.reddi.comm
VLoocaaLLaVAVommmm/////////sssss sasallaaaaeeededeeddeeeee rope rope_allows llama models to have/
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
10

---

Published as a conference paper at ICLR 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. arXiv: 2306.15595.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https: 000mss/.orrbbloog222-----3--0000 vicuna/
Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXivv1803.05457vv 2018.
Tri Dao. FlashAttention-22 -2 Faster attention with better parallelism and work partitioning. 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FltstAntention: Fast and memory-efficient exact attention with 10-awareness, 2022. arXiv:2205.141: 14135.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL http:://pp.cmmmtttttccccooooroop
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition scale, 2021.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy, The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 1xXi:2101100027, 2020.
Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, 2020. Association for Computational Linguistics.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-Infinte: Simple on-the-fly length generalization for large language models, 2023.
11

---

Published as a conference paper at ICLR 2024
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguistics, December 2020.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization, 2021.
kaiokendev. Things I'm learning while training superhot., 2023. URL httpssss//aaiooeeeeegiii.bbbbbbb ceepetteeniin--cnnteteee to-8K
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models, 2023.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, April 2020.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng. Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source Ilms truly promise on context length?, June 2023. URL https::/lmsss.oog/llg/22220231 20--0000-999---- longchat
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNIP,, 2018.
Evan Miller. Attention is off by one, 2023. URL http:///www.aammmccccc//ttttttt/ html
OpenAI. Gpt-4 technical report, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525 15344, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 12.11883/11/P111111
URL http:::/cccaathooooooo//l/111111 144
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102 2022.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables
input length extrapolation. In International Conference on Learning Representations, 2022. URL
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training 2018.
Jack W. Rae, Anna Potapenko. Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer. Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open foundation models for code, 2023.
12

---

Published as a conference paper at ICLR 2024
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint Stti::9977.10641 2019.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2I04.090 09864, 2021.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An allacc 2023.
Ilama model.
Yi Tay, Mostafa Dehghani. Dara Bahri, and Donald Metzler. Efficient transformers: survey. ACM Computing Surveys, 55(6), dec 2022. ISSN 0360-0300.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms, 2023. URL www.mossaicml..ommbbb//mmtt
Accessed: 2023-05-05.
Together.
- and tunnttnnig for llama-2 models with together api, June 32k-instruct
2023. / httpss///oogehhher////////////
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 09288, 2023b.
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. HPCA, 2021.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotporQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). 2018.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula. Qifan Wang. Li Yang, and Amr Ahmed. Big Bird: Transformers for longer sequences. In Proc. of NeurIPS, volume 33, 2020a.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. Curran Associates, Inc., 2020b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR, Abs/1995.00780 2019. URL http://arxi..og///////0000.0.000
13

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference of large language models, 2023b.
14

---

Published as a conference paper at ICLR 2024
A DISCUSSIONS
Applications. StreamingLLM is particularly suited for streaming applications, such as multi-round dialogues, where continuous operation without heavy reliance on extensive memory or historical data is crucial. For instance, in daily assistant application based on LLMs, StreamingLLM enables the model to function seamlessly over extended periods. It bases its responses on recent interactions, thus avoiding the need for frequent cache refreshes. Traditional methods might require resetting the cache when the conversation length surpasses the training length, leading to loss of recent context, or they might need to recompute key-value (KV) states from recent text history, which can be inefficient.
Limitations. While StreamingLLMM improves the efficiency of LLMs in streaming contexts, it does not extend the models' context window or enhance their long-term memory capabilities. As detailed in Section C the model is limited to operating within the confines of its current cache. Consequently, StreamingLLM is not suitable for tasks that demand long-term memory and extensive data dependency, such as long document question-answert excels in scenarios only requiring short-term memory, like daily conversations and short document QA, where its strength lies in generating coherent text from recent context without the need for cache refreshment.
(QA) and summarization. However,
Broader Societal Impacts. StreamingLLMM significantly enhances the efficiency and accessibility of LLMs, democratizing their use across various sectors. By enabling nonstop and rapid interactions in applications like conversational agents, StreamingLLM improves user experiences, especially in scenarios requiring fixed-length models. This advancement allows for more seamless and contextually aware dialogues, potentially benefiting sectors like education, healthcare, and customer service. Additionally, StreamingLLLM's efficiency in processing reduces the computational load, aligning with the need for environmentally sustainable Al technologies. This aspect is crucial in making advanced AI tools more accessible in regions with limited technological resources. However, the potential negative impacts of StreamingLLLM mirror those associated with general language models, such as misinformation and biased content generation risks. It's essential to address these risks with robust ethical guidelines and safeguards. In summary, while StreamingLLM shares some risks common to language models, its positive contributions towards enhancing user experience, democratizing Al access, and promoting sustainability are noteworthy. These benefits underscore the importance of responsible deployment and ethical use of this technology.
B ADDITIONAL RELATED WORKS
Sparse Transformers. The literature on efficient Transformer models primarily focuses on reducing the computational and memory complexity of the self-attention mechanism. A relevant line of work involves sparsifying the attention matrix by restricting the field of view to fixed, predefined
patterns, such as local windows or block patterns with fixed strides (Tay et al. 2022). Sparse Trans- former (Child et al. 2019) introduces sparse factorizations of the attention matrix, reducing the computational complexity of attention to (n) LongFormer (Beltagy et al. 2020) combines dilated local windowed attention with task-motivated global attention. Extended Transformer Con- struction Ainslie et al. (2020) presents a novel global-local attention mechanism, incorporating four types of attention patterns: global-to-globa local-to-local, local-to-giobal, and global-to-local. Building on ETC, BigBird (Zaheer et al., 2020a) proposes another linear complexity attention alter- native, utilizing global tokens, local sliding window attentions, and random attention. However, these methods have several limitations. First, Sparse Transformer and ETC require custom GPU kernels for a specific alock-sparse variant of -attrix multiplication. Second, LongFormer, ETC, and BigBird all rely on a global attention pattern, which is unsuitable for autoregressive language models. Third, these methods are incompatible with pre-trained models, necessitating retraining from scratch. In contrast, our method offers ease of implementation using standard GPU kernels and is compatible with pre-trained autoregressive language models using dense attention, which are prevalent in the NLP community. This compatibility provides significant advantage, allowing for the leveraging of existing pre-trained models without any fine-tuning.
Concurrent Works. Our research coincides with the work of Han et al.) who conducted theoretical study on the length generalization failure of language models, identifying three -fii---diittiuttion factors. Their approach, inspired by this analysis, involves employing "A" "-shaped attention pattern
15

---

Published as a conference paper at ICLR 2024
Table 7: Accuracy (in %) StreamEval with increasing query-answa distance. Each line StreamEval contains 23 tokens. Accuracies are averaged over 100 samples, and each sample contains 100 queries.
Maama-2-B-332--1m
Cache Config
Line Distances
Token Distances
4+2044
4+4092
4+8188
4+16380
20
460
85.80
84.60
81.15
77.65
40
920
80.35
83.80
81.25
77.50
60
1380
79.15
82.80
81.50
78,50
80
1840
75.30
77.15
76.40
73.80
100
2300
0.00
61.60
50.10
40.50
150
3450
0.00
68.20
58.30
38.45
200
4600
0.00
0.00
62.75
46.90
400
9200
0.00
0.00
0.00
45.70
600
13800
0.00
0.00
0.00
28.50
800
18400
0.00
0.00
0.00
0.00
1000
23000
0.00
0.00
0.00
0.00
and reconfiguring position encoding distances to enhance length generalization in LLMs. This approach bears a resemblance to our methodology. However, our work uncovers the "attention sink" phenomenon, wherein Transformer models tend to assign high attention scores to initial tokens with small semantics. This phenomenon extends beyond the scope of length generalization failure, pervasive issue in Transformer models. We observe this "attention sink" behavior not only in auto-regressive language models but also in encoder Transformers such as BERT (see Section H). and Vision Transformers (ViTs) (Darcet et al. [2023), suggesting its broader prevalence in Transformer architectures. To mitigate the "attention sink" phenomenon, we propose the introduction of learnable sink token during pre-training, and we support findings with extensive ablation studies.
indicating
In parallel. Darcet al. observed similar attention concentration on random background patch tokens in Vision Transformers, termed as "registers." These registers act as repositories for global image information. Their solution, adding dedicated "register" tokens, aims balance attention distribution. Our finding of "attention sinks" parallels this concept. In our paper, the "attention sinks" are initial tokens that disproportionately attract attention from subsequent tokens. Introducing dedicated sink token during pre-training prevents the model from inappropriately using content tokens as attention sinks, leading to more effective attention distribution. However, key difference exists: "registers" in Vision Transformers function global information holders within intermediate layers, whereas "attention sinks" " are positioned as initial tokens autoregressive models. This positional variance suggests that the softmax function in attention computation might play more fundamental role in the emergence of attention sinks.
C ACCURACY ON STREAMEVAL WITH INCREASING QUERY-ANSWER LINE DISTANCE
To StreamingLLM's handling of extended inputs, we evaluated the Llama-2-7B-322 2-1-7BB-22K-nnstnnee model on StreamEval, focusing on different query-answer line distances under various cache con- figurations. In StreamEval, each line consists of 23 tokens, making the line distances equivalent to token distances of 23 x line distances. Accuracy was calculated by averaging results over 100 samples, with each sample comprising 100 queries. Table illustrates that StreamingI.LMM retains accuracy when the token distance between the query and answer is within the cache size. However, accuracy diminishes as this distance increases and eventually drops to zero when it surpasses the cache capacity.
These results demonstrate that while StreamingL is effective in generating coherent text based on recent context, it cannot extend the context length of language models. These results also emphasize a broader challenge in current language models: their inability to fully utilize context information within the cache, finding that aligns with the observations made by Liu et al.
16

---

Published as a conference paper at ICLR 2024
Table 8: Performance comparison of StreamingLLM against the default truncation baseline LongBench (Bai et al. 2023). The baseline truncates inputs to 1750 initial and 1750 final tokens. StreamingLLM. LLM 4+3496 uses 4 attention sink tokens and 3496 recent tokens, while StreamingLLM 1750+1750 uses 1750 tokens for both initial and recent segments.
Single-Document QA
Multi-Docunnnn QA
Summarization GovReport MultiNews
Llama2-7B-chat
NarrativeQA Qasper
HotpotQA 2WikiMQA
Truncation 1750+1750
18.7
19.2
25.4
32.8
27.3
25.8
LL.eamingLLLM 443496
11.6
16.9
21.6
28.2
23.9
25.5
StreamingLLM 1750+1750
18.2
19.7
24.9
32.0
26.3
25.9
D LONG-RANGE BENCHMARK EVALUATION
We evaluated Streaming LLM using the Llama-2- 2-7B chat model (max context length 4k) Long- Bench (Bai et al., (023). which encompasses three key NLP tasks: single-document QA (Narra- tiveQA (Kočiský ||| and Qasper (Dasigi al.] 2021)) )), multi-document QA (HotpotQA (Yang)
et al. 2018) and 2WiKIMQA Ho et al. (2020)) and summarization (GovReport (Huang et al. 2021
MultiNews (Fabbri et al., 2019)) LongBench sets default max sequence length of 3,500 tokens for the Llama-2-7B-chal model, truncating from the middle to preserve beginning and end information (1,750 tokens each). Table shows that StreamingLLM with 4+3496 cache configuration under- performs compared to the truncation baseline, likely due to the loss of crucial initial input prompt information. However, aligning the attention sink number to 1750 restores performance the level of the text truncation baseline. These results corroborate the findings in Section C. demonstrating that Streamingl. effectiveness is contingent on the information within its cache, with in-cache performance comparable to the text truncation baseline.
E LLAMA-2-7B ATTENTION VISUALIZATION ON LONGER SEQUENCES
modelaavers.do D attn head
mode: layers 6.seff atin head
model loyers 12.slf attn head
70
40
2.00
40
2
3.50
30
1.2%
1000
100
100
-
120
120
(a) Layer 0 Head 0
(b) Layer 6 Head 0
(c) Layer 12 Head 0
model lovers 18 sett attn head
mocel Loyers 28l attn head
mocel loyers 30.eeft attn head
20
40
100
220
120
-6 120
(d) Layer 18 Head
(e) Layer Head
(1) Layer 30 Head
Figure 11: Visualization of the average attention logits in Llama-2-7B over 256 sentences, each with length of 128.
17

---

Published as a conference paper at ICLR 2024
Liama-2-7B Attention Score the First Token in Each Layer (SeqLen = 4096)
1.0
Attention Score on the First Token
0.8
0.6
0.4
0.2
0.0
0
10
15
20
25
30
Layer ID
Figure 12: Visualization attention scores (after SoftMax) on the first token across layers Llama- 2-7B. Attention Scores the 4096th token's 's attention towards the first token in each layer. The error bars are the standard deviation the first token's attention scores across different heads one layer. Results averaged over 256 sentences, each having length of 4096 tokens.
Figure visualizes the attention map of Llama-2-7B using short sequences (length 16) clarity. We further visualize the attention of Llama-2-7B on longer sequences (length of 128) in Figure Ti] find the observations on short sequences also hold on longer sequences, where the attention scores of the initial tokens much higher than the rest of the tokens in most layers, regardless of the distance between the initial tokens and the tokens the rest of the sequence. Because the longer the sequence, the thinner the attention sinks' scores visualized the heatmap. We further analyze the attention distribution on longer sequences (length 4096) using a different method in Section
F QUATITATIVE ANALYSIS OF ATTENTION SINKS IN LONG INPUTS
Figures 2 and 13 illustrate the attention sink phenomenon using short sequences for clarity. Extending this analysis, Figure 12 demonstrates the distribution of attention scores (after SoftMax) towards the first token in lengthy inputs (sequence length of 4096). We average attention scores across 256 sequences, with each sequence comprising 4096 tokens. The plotted data represent the attention allocated by the 4096th token to the initial token in every layer. Notably, the attention scores for the first token significantly high, often exceeding half of the total attention, except for the two bottom layers. This observation empirically substantiates the preferential focus on the first token by the majority layers and heads, irrespective of other tokens' distances within the sequence. Such a trend underscores the critical role of the initial tokens in sequence, as their removal has huge impact on language model performance due large portion of the denominator in the SoftMax function being removed.
G 1LAMA-2-708 ATTENTION VISUALIZATION
Figure 2 shows the attention visualization -lama-2-7B, we further visualize the attention of Ilama-2 Llama-2-70B in Figure 13 We find the observation Llama-2-7B also holds Llama-2-70B,
18

---

Published as a conference paper at ICLR 2024
mod Agara 2st herd
100
(a) Layer Head
(b) Layer 0 Head
(c) Layer 8 Head 0
(d) Layer 8 Head
Morl stribedt
30
2
10
(e) Layer 16 Head 0
(l) Layer 16 Head
(g) Layer 24 Head 0
(h) Layer 24 Head
vadelaayer or bost
macelbpee trael arohwadt
thsee arr/waad
rede Mebtbyeeeeoo tooe
30
(i) Layer 32 Head 0
Layer 32 Head
(k) Layer 40 Head 0
(1) Layer 40 Head
ood daattttt hed
nocel lours 4tuet atrieed
mace tovn Stae stmieed
rood wenttatt racc
(m) Layer 48 Head uiv 2os
(n) Layer 48 Head
(0) Layer 56 Head 0
(p) Layer 56 Head
bonc strel atntess
bean 2hcet atniwadd
rede byen rean
-2
(g) Layer Head
(r) Layer 64 Head
(s) Layer 72 Head 0
(1) Layer Head
Figure 13: Visualization of the average attention logits in Llama-2-70B over 256 sentences, each with length of 16.
19

---

Published as a conference paper at ICLR 2024
where the attention scores of the initial tokens much higher than the rest of the tokens in most layers.
H ATTENTION SINKS IN ENCODER TRANSFORMERS
Laver
head
Layer 0
(C65)
sheaming
0200
6:175
02 PEaWTkI
atccerry
0.000
morf
Layer head 0
Laver
headd
Jusa
sam
64
loc
cre
03
Papu
Layer
[CLS
(CLO) SONMEIIN
Ant
co
cn
on
toes
tost
cn
pofomane
POR
Pogu
ICLS
ICUS
Layer
ICLSI
Layer
con
ow
IC
length
#dpromann
on
office crc
0.0
L000
Figure 14: Visualization of attention maps for sentence "StreamingLLM work infinite-length texts without compromising efficiency and performance." SERT-base-uncasee
In this paper, we mainly explore the attention sink phenomenon observed in autoregressive, decoder- only language models like GPT and Llama. Building upon the insights from Section 3.1 propose that this phenomenon likely extends to other Transformer architectures, including encoder models such BERT (Devlin et al.] 2019) and ViT (Dosovitskiy et al.] 2021). This assumption stems from the fact that these models share a similar Transformer structure and utilize SoftMax attention mechanisms. To substantiate hypothesis, analyze the attention patterns of BERT-base unse-usdddd
20

---

Published as a conference paper at ICLR 2024
Table 10: Comparison of vanilla attention with prepending zero token and learnable sink token during pre-training. Cache config zY denotes adding initial tokens with recent tokens. Perplexity evaluated on the first sample in the PG19 test set.
Cache Config
0+1024
1+1023
2+10022
4+1020
Vanilla
27.87
18.49
18.05
18.05
Sink Token
1235
18.01
18.01
18.02
+ Sink Tokens
1262
25.73
18.05
18.05
as depicted in Figure 14 Our findings reveal that BEIRT-base-uncseed exhibits the attention sink phenomenon, characterized by disproportionatly high attention scores assigned to the [SEP] ] token in most layers. This indicates that the model consistently relies on the omnipresent [SRP] token as focal point for attention. Furthermore, concurrent research by Darcet et al. identifies similar attention spikes in Vision Transformers, attributed to random background patch tokens acting "registers" for global image information. We contend that these "registers' are analogous to the attention sink phenomenon observed, suggesting that this is universal characteristic across all Transformer models.
I USING MORE SINK TOKENS IN THE PRE-TRAININ STAGE
Section 3.3 illustrated that incorporating single dedicated sink token in the pre-training stage doesn't t affect model performance but enhances streaming performance by centralizing attention sinks to one token. This section delves into whether adding additional sink tokens during pre-training could further optimize the performance of pre-trained language models.
As depicted in Figure 15, our experiments show that incorporating either one or two sink tokens during pre-training results in pre-training loss curves that closely resemble those of the baseline (vanilla) model. However, as detailed in Table 9 the introduction of second sink token does not yield substantial improvements performance across most benchmark tasks.
Further analysis, as shown in Table 10 reveals that the inclusion of additional sink tokens does not enhance streaming performance. Interestingly, the model appears to rely on both sink tokens to maintain stable streaming performance. These findings suggest that while single sink token is adequate for improving streaming performance, adding more sink tokens does not lead to further enhancements in overall language model performance. This contrasts with findings in Vision Transformers (ViT) (Darcet et al. 2023), where multiple "registers" have been found to beneficial.
2.0
Table 9: Zero-shot accuracy (in %) across NLP benchmarks, including ARC-[Challenge, Easy], HellaSwag, LAMBADA, Open- bookQA, PIQA, and Winogrande.
Vanilla Sink Toxan
2.7
Sinic
2.6
Methods
ARC-c
ARC-
HS
LBD
OBQA
PIQA
WG
20
40
Staps
80
100
120
140
Vanilla
18.6
45.2
29.4
39.6
16.0
62.2
50.1
+ Sink Token
19.6
45.6
29.8
39.9
16.6
62.6
Pre-training loss curves models with 0, and 2 sink tokens.
Figure 15:
+ 2 Sink Tokens
18.7
45.6
29.6
37.5
15.8
64.3
50.4
21


Reviews:
[{"review_id": "wphsValytr", "reviewer": "Reviewer_PzfP", "summary": "This work proposes the addition of a placeholder token, called a token sink, to language models.This is motivated by the observation that significant attention mass is placed at the initial token positions in a sequence and performance deteriotes whenever tokens in this position are dropped/evicted. The authors carry out extensive experiments to validate this claim and make a convincing argument for the placement of a special token in the initial position, whose purpose is to help regulate and channel the attention mass across layers. This leads to stable language model performance at million-scale sequence lengths.", "questions": "It is interesting to see how concentrated attention gets on the initial tokens as you go deeper into the model, could it be said that these tokens are carrying out some form of global memory/attention operation?\n\nHow does this work relate to [Vision Transformers Need Register](https://arxiv.org/abs/2309.16588v1).. This statement from this work reminds me of the registers paper; \"As a result,\ninitial tokens are more easily trained to serve as attention sinks, capturing unnecessary attention\"\n\nLooking at the StreamEval benchmark, it would also be good to have an idea of how performance decays with length. How does it perform when you ask it a question related to line 1k, after it has seen 10/100k lines. This gives a better idea of long-context/sequence performance profile than consistently quering only 20 lines prior. I would be willing to improve my score if i had better insight of how this method performed on such a benchmark over (extremely) long horizons. How it performs relative to the evicted tokens.\n\n\nUpdate: Authors addressed all my questions.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, the problem is well motivated with many experiments used to help make a convincing claim.\n\nAll code and datasets are made available, allowing for easy reproducibility and validation of claims made in this work.", "weaknesses": "The claims on long horizon performance need additional clarity and evaluation. I am not sure how practically useful this approach is for long-horizon language modelling.. For example, at million-scale sequence lengths, when intermediate tokens are evicted, how does performance translate relative to the evicted tokens? Doesnt it mean, the model is only practically useful for information covered at the beginning of the sequence and at the end?\n\nThere is a missing section on the broader societal impacts of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dv0vEJdMrF", "reviewer": "Reviewer_obg3", "summary": "The paper focuses on the challenge of efficiently scaling and generalizing beyond the training sequence length for large language models. Specifically, they propose to cache and attend to the initial tokens of the sequence in addition to the most recent tokens used in window attention. The main motivation behind this proposal is the \"attention sink\" phenomenon that is caused by the strong attention scores toward initial tokens that are accumulated as the sequence increases. Since this method introduces this simple change of attending to the initial tokens, it can be applied to any LLM; they also show it is possible to train the model with a single token as attention sink for further efficiency. Experiments on a models of vaying model size show that it allows them to generalize far beyond their context window and it is much faster than sliding window with recomputation.", "questions": "- Have you considered evaluating on a benchmark for long-range tasks such as LongBenh, MuLD or Long-range arena? It would help quantify limitations of the proposed attention in terms of context utilization.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Several methods have been introduced in the past for generalizing to longer context lengths but most of them require training the model from scratch or through continued training. The observation that this paper is making about the attention sink phenomenon is fascinating and inspires a simple solution that requires no additional training that prior work has overlooked. \n- The paper is positioned well wrt. prior length generalization work and shows how different relative positional embedding methods such as RoPE anD Alibi can be combined with attention sinks. The fact that it can be combined with any of these methods with minimal effort makes it potentially very impactful; virtually any transformer model can be modified to handle large context sizes in this way. \n - The exposition is clear and experimental part well executed. The empirical analysis based on actual attention logits and the theoretical explanation of why it happens is convincing and intuitive. \n- Evaluation covers models of different sizes and different metrics (perplexity and downstream performance) and shows that proposed attention is effective across all sequence lengths.", "weaknesses": "- The proposed attention is reminiscent of methods that introduce sparse attention patterns such as Sparse Transformer but there is no in-depth discussion that draws a connection.\n- Even though attention sinks maintains the perplexity levels in check for extremely large sequence lengths, the paper does not study in detail on how good utilization of the context the model is doing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XyOtZijfsr", "reviewer": "Reviewer_STcB", "summary": "This paper demonstrates that the first few token(s), called attention sink, take most of the attention scores in large language models and propose StreamingLLM by keeping them in the KV cache. StreamingLLM achieves significant speedup compared to the baseline of sliding window attention with recomputation while having reasonable language modeling perplexity and zero-shot accuracy on several NLP benchmarks. Additionally, the authors pre-trained a small language model with a single learnable sink token so that StreamingLLM can only use one token for attention sink.", "questions": "Could you elaborate more on why (1) window attention blows up and (2) attention sink exists?\nI think current explanations are mostly empirical and not theoretically grounded. Specifically, I agree that attention scores could be skewed toward earlier tokens due to the autoregressive nature of language models. However, why very few (less than 4) tokens are enough to cover most attention scores?\n\nVisualization on a short sequence (length 16) in Figure 2 might not be enough. Does the same phenomenon occur for longer sequences, which is our main interest? I guess some important tokens with high attention scores might be shown up for each text (of course, it will not be shown if you average per position over different texts). Do you have any quantitative values on how much attention sink occupies the attention scores (especially for longer inputs)?\n\nIn Figure 3, MPT has different patterns from other LLMs. Is it because MPT uses ALiBi and others use RoPE? Any explanation?\n\nRegarding pre-training with a sink token, I think reducing 4 -> 1 is not important considering the large size of the cache. Am I missing? It could be marginal but why is a model with a sink token consistently better than the vanilla one? An LM with 160M parameters is used, and I understand this is because of the academic budget. I am just wondering if the same conclusion can be guaranteed for larger LMs. \n\nIn Figure 10,  why do two options have different memory usage for 4096 cache size?\n\nThis paper uses the attention sink for efficient streaming of LLMs. However, my one aspect is having attention sink is a kind of artifact of autoregressive LLMs not fully utilizing its representation power. Can we make/train LLMs not to have the attention sink for better performance?\n\nThere is almost similar concurrent work: LM-Infinite (https://arxiv.org/abs/2308.16137). It is not required, but how do you compare your work with this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Improving the efficiency of language modeling with long text input is a practically important and very timely topic. The paper is well motivated and structured in that it first introduces problems, suggests an easy and effective solution, and validates it with experiments and required ablation studies.", "weaknesses": "StreamingLLM is compared with only very basic baselines (i.e., (a) dense attention, (b) window attention, and (c) sliding window with re-computation) without alternative ways to use LLMs with longer inputs. The authors argue StreamingLLM’s 22.2x speedup compared to (c), but it looks too extreme because there could be a sweet spot that could achieve reasonable ppl and inference speed by sliding window with appropriate strides (less frequent re-computation instead of every step).\n\nThe paper argues that StreamingLLM can handle infinite sequence length, but again the abovementioned basic baselines on PG-19 are used to show this. Experiments on benchmark datasets that truly require long context modeling are necessary. Also, providing numbers of SoTA models on PG-19 could be helpful in understanding how absolutely StreamingLLM performs well, though accuracy might not be the main interest of this paper. Particularly, using a small size cache (not beyond the training sequence length) makes it suspicious whether StreamingLLM is capturing long context information well. I guess it will have difficulty utilizing information from tokens that do not belong to the cache at the prediction step, and this can be easily checked with synthetic datasets (e.g., Little Retrieval Test).\n\nThere have been several previous works that observe position bias on attention patterns in transformers. Also, there are existing architectures, including LongFormer, BigBird, and ETC, that have global tokens for attention, though most of them are encoder-based language models. It is worth having a discussion on them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgsYBaMhW6", "reviewer": "Reviewer_MUgn", "summary": "The paper introduces StreamingLLM, a framework that enables large language models (LLMs) to handle infinite sequence lengths without fine-tuning. The paper makes the following contributions:\n\n- It observes the attention sink phenomenon, which means that LLMs tend to allocate a lot of attention to the initial tokens, regardless of their semantic importance.\n- It proposes the StreamingLLM framework, which leverages the attention sink phenomenon to retain only the most recent tokens and a few attention sinks, discarding intermediate tokens. This allows LLMs to generate coherent text from recent tokens without a cache reset.\n- It suggests adding a placeholder token as a dedicated attention sink during pre-training, which can further improve the streaming performance of LLMs.\n- It demonstrates that StreamingLLM can enable models like Llama-2, MPT, Falcon, and Pythia to perform efficient language modeling with up to 4 million tokens.", "questions": "- The authors focus on LLMs that are autoregressive, meaning that they generate text one token at a time, conditioned on the previous tokens. Dose the attention sink phenomenon exist in other types of models, such as bidirectional models (e.g., BERT) or non-transformers models?\n- Can attention sink be applied to the pre-training stage, retaining more placeholders from scratch to further improve the performance?\n- Does this phenomenon also exist in larger models like Falcon-40B, LLaMa2-70B?\n- Does the Group Query Attention structure have any impact?\n- If you combine proposed Attention Sink with Sliding Window w re-computation, I wonder whether it performs better than re-computation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper studies an important topic/setting on the long context of large language models, how to enhance LLMs to handle long/infinite sequence without fine-tuning.\n- Attention sink phenomenon is remarkably interesting. It was found that retaining the KV of initial tokens can significantly improve the performance of window attention. \n- This paper poses comprehensive experimental investigations on different large language models, and it also shows that StreamingLLM outperforms other baselines by up to 22.2× speedup in streaming settings.\n- The paper is well written, clear to follow.", "weaknesses": "See Questions Section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)