SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A fixed memory budget for one attention head is  $B$  tokens. In other words, we can store key and value embedding for  $B$  previous tokens. We describe the problem as follows,
Definition 4.1 (Sequential generation at an attention head under budget  $B$ ). Given a stream of token embedding, including prompt and previously generated tokens, denotes their input to the head as  $\{x_1, \ldots, x_t, \ldots\}$ . The problem of sequential generation at an attention head under budget  $B$  is maintaining a key cache  $\bar{\mathcal{K}}_t$  and value cache  $\bar{\mathcal{V}}_t$  such that  $\bar{\mathcal{K}}_t, \bar{\mathcal{V}}_t \in R^{n \times d}$  and  $n < B$ .
Approach. Inspired by the textbook solution of reservoir sampling and the Least Recent Usage cache replacement algorithm, SCISSORHANDS reserves a fixed memory buffer for the KV cache. When the buffer is full, SCISSORHANDS drops stored but non-influential tokens from the cache. We present the main algorithm in Algorithm 1 and Algorithm 2.
When the KV cache size exceeds the budget, SCISSORHANDS drops tokens from the KV cache according to Algorithm 2. The importance record is a counter that indicates how many times a token is deemed non-important. We choose attention scores as the importance indicators, following our methodology in Section 3.2. The importance record is collected over a history window  $w$  to reduce variance. A higher counter suggests dropping from the cache. Recent tokens are always kept because of the lack of information on their importance by setting the counter for all tokens in the recent window  $r$  to 0.
With a sampled KV cache, attention output can be computed by the following estimator
$$
\hat {a} _ {t} = \sum_ {i = 1} ^ {n} \hat {\alpha} _ {t, i} \bar {\mathcal {V}} _ {t} [ i ], \text {w h e r e} \hat {\alpha} _ {t, i} = \frac {\exp (\langle x _ {t} W _ {Q} , \bar {\mathcal {K}} _ {t} [ i ] \rangle)}{\sum_ {i = 1} ^ {n} \exp (\langle x _ {t} W _ {Q} , \bar {\mathcal {K}} _ {t} [ i ] \rangle)}
$$
Overhead Tradeoff At the compression step, an extra attention computation is introduced to collect the importance measurements over a history window. However, such compression is not required at every generation step.  $m$  controls the frequency, and we use  $m = 0.5B$  in our experiment. Further, steps after the compression have reduced attention computation because of the reduction in the KV cache. On the other hand, one can trade a tiny amount of memory to avoid the overhead by maintaining the importance record during generation steps in Algorithm 1.
Allocating Budgets Across Attention Heads. An LLM typically consists of  $L$  transformer layers where each layer has  $H$  heads. A total memory budget has to be distributed over layers and heads. Within each transformer layer, the budget is distributed evenly across heads. Within the entire model, we distributed the budget according to Figure 2. The rule of thumb is to allocate more budget to later layers to compensate for the lower persistence ratio.
4.2 Theoretical Analysis.
We study how much the tokens generated by the compressed KV cache deviate from the tokens generated by the original transformer using our simplified model in (1). Let  $\{\tilde{x}_t\}_{t=0}^T$  denote the tokens generated by the transformer with budget KV cache as in Algorithm 2 with  $m = 1$ :
$$
\tilde {x} _ {t + 1} = \mathcal {F} \left(\tilde {a} _ {t}\right), \text {w h e r e} \tilde {a} _ {t} = \operatorname {s o f t m a x} \left(1 / _ {t} \cdot \tilde {x} _ {t} W _ {Q} \tilde {\mathcal {K}} _ {t} ^ {\top}\right) \tilde {\mathcal {V}} _ {t} ^ {\top} W _ {O}
$$
Notice that when  $m = 1$ , i.e., in each iteration, we drop one token with the lowest score, the cache will always maintain  $B$  tokens. If the ranking of the attention scores does not change in each iteration, Algorithm 2 will always drop tokens with the smallest attention scores.
For reference purposes, let  $\{x_{t}\}_{t = 0}^{T}$  denote the tokens generated by a vanilla transformer defined in (1). We will bound the difference  $\| x_{t} - \tilde{x}_{t}\|_{2}$
Theorem 4.1. Let  $\lambda_1, \lambda_2$  denote the largest singular values of  $W_1$  and  $W_2$  in (2). Let
$$
\beta_ {t, j} = \frac {\exp \left(^ {1 / t} \cdot \tilde {x} _ {t} W _ {Q} W _ {K} ^ {\top} \tilde {x} _ {j} ^ {\top}\right)}{\sum_ {i = 1} ^ {t - 1} \exp \left(^ {1 / t} \cdot \tilde {x} _ {t} W _ {Q} W _ {K} ^ {\top} \tilde {x} _ {i} ^ {\top}\right)}
$$
and assume that each  $\beta_{t,j} = cv_{t,j}$ , where  $v_{t,j}$  are sampled from a power-law distribution with pdf  $f(x) = c(x + b)^{-k}$ . Suppose that  $\lambda_V\lambda_O(1 + \lambda_1\lambda_2)(1 + \lambda_Q\lambda_K)\leq \frac{1}{2}$ . Let  $T_{\mathrm{min}}$  and  $T_{\mathrm{max}}$  denote the starting and maximum sequence lengths, respectively, and let  $B\le T_{\mathrm{max}}$  denote the budget as in Algorithm 2. If for all  $t\in [T_{\mathrm{min}},T_{\mathrm{max}}]$ ,  $S_{t}$  contains only tokens with at most the largest  $B$  values of  $\beta_{t,j}$ , that is,  $|S_t| = B$  and  $\min_{j\in S_t}\beta_{t,j}\geq \max_{j\notin \hat{S}_t}\beta_{t,j}$ , then for all  $\epsilon \in (0,1)$ , with probability at least  $1 - T_{\mathrm{max}}\exp \left(-\frac{\epsilon^2b^2(T_{\mathrm{min}} - 1)}{(k - 2)^2(u - b)^2}\right) - T_{\mathrm{max}}\exp \left(-\frac{2(T_{\mathrm{min}} - 1)(1 - B / T_{\mathrm{max}})^2}{(1 - \epsilon)^2}\right)$ , the following error bound must hold for all  $t\in [T_{\mathrm{min}},T_{\mathrm{max}}]$
$$
\mathbb {E} \left[ \| x _ {t} - \tilde {x} _ {t} \| _ {2} \right] \leq \frac {2 . 1 \left(1 - ^ {B} / T _ {\max }\right)}{(1 - \epsilon) ^ {2}} \left(k - (k - 1) \left(\frac {1 - \epsilon}{B / T _ {\max } - \epsilon}\right) ^ {1 / (k - 1)}\right) \tag {4}
$$
The definition of  $\beta_{t,j}$  means the attention scores computed on the tokens generated by the compressed approach. Our theorem assumes that dropping the tokens depends on the attention score of the current iteration. (4) provided a bound on the expected difference between the tokens generated in the budget and the original approach. The upper bound scales with  $1 - B / T_{\mathrm{max}}$ . When  $B = T_{\mathrm{max}}$ , meaning that we are keeping all of the tokens, the error becomes zero. The term  $k - (k - 1)\left(\frac{1 - \epsilon}{B - \epsilon}\right)^{1 / (k - 1)}$  depends on the distribution that the attention scores are fitted to and is always less than one. With a strong power-law distribution, this term provides a further decrease to the error bound in (4).
In this section, we present the results that demonstrate SCISSORHANDS achieves up to  $5 \times$  reduction in the KV cache memory compared to the standard model with no accuracy loss. We also show that SCISSORHANDS is compatible with 4-bit quantization.
Experiment Setting. We compare the accuracy of SCISSORHANDS-OPT against the original OPT on one language model datasets C4 [23] and a number of few-shot downstream tasks: Hellaswag [27], MathQA [28], PIQA [29], Winogrande [30]. We use lm-eval-harness [31] to evaluate few-shot tasks. Our experiments are conducted on NVIDIA 4 A100 40GB GPU servers.
No Accuracy Drop until  $5 \times$ . In Figure 3, we present SCISSORHANDS's accuracy trend where  $1 \times$  denotes the original OPT. In the language modeling setting, perplexity is the lower the better. For OPT-6B, perplexity is maintained until  $50\%$  of the original KV cache size for OPT-13B. For OPT-66B, perplexity is maintained until  $75\%$  of the original KV cache. We observe a flatter accuracy trend as the model size grows, which is exceptionally encouraging. This suggests that SCISSORHANDS can scale with the model size. Downstream tasks are usually less sensitive to perturbation and bear more variance in terms of accuracy. We evaluate the 5-shot setting and  $1 \times$  denotes the original OPT model. For Winogrande and MathQA, accuracy is maintained even after  $5 \times$  compression for OPT-66B. Similar to the language modeling setting, SCISSORHANDS performs better at larger models. Generally, accuracy is maintained with  $15\% - 30\%$  of the original KV cache size.
(a) Language Modeling
(b) OPT-6B Five shot
(c) OPT-13B Five shot
(d) OPT-30B Five shot
Figure 3: Accuracy trend of SCISSORHANDS on language modeling dataset and downstream tasks with different KV cache compression. In general, SCISSORHANDS incurs no accuracy drop until  $5 \times$  compression on OPT-66B.
Ablation on the Importance of Pivotal Tokens We divide C4 into three subsets depending on the sequence length. C4-[256-512] contains data sequences that are longer than 256 tokens but less than 512 tokens. C4-[512-1024] contains data sequences longer than 512 tokens but less than 1024 tokens. C4-[1024-2048] contains data sequences that are longer than 1024 tokens but less than 2048. Results are summarized in Table 3. Local Windows refers to only keeping tokens in the recent window, while SCISSORHANDSkeeps both recent tokens and pivotal tokens. We observe the perplexity of the full model degrades slightly with the growing sequence length. At all sequence lengths, SCISSORHANDS's performance is comparable against the full cache model, while Local Window incurs a significant quality loss. This demonstrates that keeping the pivotal tokens is important to reserve model performance. It is also interesting to note that at longer sequence lengths, the local window has higher accuracy. This also shows at longer sequence length, the attention mechanism in current architecture tends to focus on recent context.
Table 3: Perplexity on C4 with different sequence lengths.
[256 - 512]	[512- 1024]	[1024- 2048]
OPT-13B	8.7968	9.1017	9.3005
OPT-13B + Local Window	81.8297	29.3823	15.5883
OPT-13B + SCISSORHANDS	8.7972	9.1011	9.3009
Table 4: Applying 4-bit quantization on top of SCISSORHANDS on Hellaswag.
OPT-6B
Original	SCISSORHANDS	SCISSORHANDS+ 4-bit
0.702	0.706	0.704
OPT-13B
Original	SCISSORHANDS	SCISSORHANDS+ 4-bit
0.720	0.720	0.720
Compatible with 4-bit Quantization We test the compatibility of quantization and SCISSORHANDS at  $2\times$  compression. We adopt 4-bit quantization following [7]. Even Hellaswag is most sensitive based on Figure 3, adding quantization doesn't introduce compounded errors.
Ablation on Attention Score Error. We present the change ratio in attention score between original OPT-13B and SCISSORHANDS OPT-13B at  $3\times$  compression on C4 in Figure 4.
We observe the attention score generated from SCISSORHANDS is almost the same as the original KV cache, which also echoes Theorem 4.1. The change ratio is calculated as  $\frac{\alpha_s - \alpha_o}{\alpha_o}$  where  $\alpha_{s}$  is the SCISSORHANDS attention score and  $\alpha_{o}$  is the original score. From Figure 4, we observe that the change ratio is centered around 0. -1 indicating that  $\alpha_{s}$  is significantly smaller compared to the original, suggesting that a small portion of the important tokens are dropped in the cache. To explain the above observation of SCISSORHANDS, we denote the  $n$  number of tokens with the highest score as  $\{x_t^{top\_ n}\}_{t = 0}^T$ . Then, for any other sets of tokens  $\{x_t'\}_{t = 0}^T$  that has no greater than  $n$  tokens, we can easily prove that similarity  $\left(x_t^{topB}, x_t\right) \leq (x_t', x_t)$ . Thus, SCISSORHANDS gives the most similar output as the original model at all layers.
6 Discussion, Limitation, and Future Work
We discover repetitive attention patterns given trained language models. One interesting question that needs to be answered is whether such behavior is a model architecture bias or an unexpected training outcome. For such purpose, we perform the same experiment with a randomly initialized OPT, and compare it against the results presented in Section 3.1. As shown in Figure 5, the repetitive attention pattern does not exist in randomly initialized models. Apart from an efficiency deployment perspective, could such repetitive attention patterns contribute to some known problems in language generation, such as repetitions? It may be
worth investigating the relationship between repetitive attention patterns and undesired generations.
Figure 4: Score between OPT and SCISSORHANDS.
Due to the limitation of the server in academics, the largest model we can fit is OPT-66B. We try to understand the behavior and verify the generality across the different models and datasets. However, we cannot access the training process and fail to know exactly how an LLM is trained to exhibit such behavior. Experiments with the large model create carbon dioxide emissions. However, our work improves the efficiency of LLM, and we foresee no negative impacts.
(a) Attention map of the token at position 178
(b) Attention map of the token at position 228
Figure 5: We plot the attention map corresponding to Section 3.1 but with a randomly initialized OPT. We observe no repetitive attention for a randomly initialized model.
Inspired by our intriguing findings that pivotal tokens exert a lasting influence on future steps, we developed SCISSORHANDS to leverage this observation to reduce the memory usage of KV cache. Our method achieves memory reductions of  $5 \times$  in the KV cache without compromising the performance of LLMs. Furthermore, we demonstrate the compatibility of SCISSORHANDS with quantization techniques, opening up the possibility of reducing memory usage in both the representation and sequence length dimensions.
We would like to thank the anonymous reviewers for their helpful discussions and feedback. This work is supported by NSF-CCS-2211815, ONR-DURIP and NSF-BIGDATA-1838177.
[18] Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Re. Mongoose: A learnable lsh framework for efficient neural network training. In International Conference on Learning Representations, 2021.
[19] Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413-17426, 2021.
[20] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[21] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
[22] Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019.
[23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
[24] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[25] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
[26] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[27] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[29] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piaq: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[30] Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin. Winogrande: An adversarial winograd schema challenge at scale. In Communications of the ACM, 2019.
[31] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. In Version v0.0.1. Sept. Zenodo, September 2021.
A.1 Repetitive Attention Pattern
We provide the attention map similar to Figure 1 but from a different transformer layer on the same text in Figure 6, Figure 7, Figure 8 and Figure 9. A repetitive pattern and attention sparsity can be observed across layers.
(b) Attention map at position 228
(c) Attention map at position 278
Figure 6: Attention Map at Layer 5
(a) Attention map at position 178
(c) Attention map at position 278
Figure 7: Attention Map at Layer 10
Figure 8: Attention Map at Layer 15
(a) Attention map at position 178
(c) Attention map at position 278
Figure 9: Attention Map at Layer 20
A.2 Cross Layer Cosine Similarity
In Section 3.3, our analysis assumes a large cosine similarity between the input and output of  $\mathcal{F}$ . Here, we provide empirical evidence to support such an assumption in Figure 10. Because of the residual connection in  $\mathcal{F}$  and the domination of  $x$ , the cosine similarity between  $x$  and  $\mathcal{F}(x)$  is extremely high.
A.3 Generated examples with SCISSORHANDS
(a) Cosine Similarity
(b) Norm Comparison
Figure 10:  $x$  and  $\mathcal{F}(x)$  is high in cosine similarity
Table 5: Generated examples using OPT-13B with full cache and SCISSORHANDSat different compression ratio.
Prompt	Hi. Does Asus ever plan to update the intel and nvidia display drivers on the UL30VT laptop? The only downloadable drivers from the Asus website are at least 3 years behind and new features for both graphics chips have been released since then. I sent a message to Asus but no response was given. Intel and Nvidia will not allow the user to download the new drivers directly. problem is the drivers won’t install. states something about your notebook needs custom drivers from the manufacturer due to customized this that or the other.
OPT-13B	I have the same problem. I have a laptop with an Intel HD Graphics 4000 and an Nvidia GeForce GT 740M. I can’t install the drivers from the Asus website. I have to use the drivers from the Nvidia website.
3x Compression	I have the same problem. I have a laptop with an Intel HD Graphics 4000 and an Nvidia GeForce GT 740M. I can’t install the drivers from the Asus website. I have to use the drivers from the Nvidia website.
6x Compression	I have the same problem. I have a custom laptop and I can’t install the drivers.
We consider the token generation process of a simplified model: a single-layer transformer model with single-head attention.
$$
x _ {t + 1} = \mathcal {F} \left(a _ {t}\right), \text {w h e r e} a _ {t} = \operatorname {s o f t m a x} \left(1 / t \cdot x _ {t} W _ {Q} W _ {K} ^ {\top} X _ {t - 1} ^ {\top}\right) X _ {t - 1} W _ {V} W _ {O} \tag {5}
$$
$x_{t} \in \mathbb{R}^{1 \times d}$  is a row vector.  $X_{t-1} \in \mathbb{R}^{(t-1) \times d}$  denotes the aggregation of  $x_{1}, \ldots, x_{t-1}$ , where the  $j$ th row is  $x_{j}$ .  $W_{Q}, W_{K}, W_{V} \in \mathbb{R}^{d \times p}$  and  $W_{O} \in \mathbb{R}^{p \times d}$  are the attention weights. Lastly,  $\mathcal{F}: \mathbb{R}^{1 \times d} \to \mathbb{R}^{1 \times d}$  denotes the MLP block following attention block, a two-layer MLP with skip connections, given by
$$
\mathcal {F} (x) = x + W _ {2} \operatorname {r e l u} \left(W _ {1} x\right) \tag {6}
$$
We are interested in the attention scores  $\alpha_{t} = \mathrm{softmax}(1 / t\cdot x_{t}W_{Q}W_{K}^{\top}X_{t - 1}^{\top})$ . Notice that  $\alpha_{t,j}$  scales with  $x_{t}W_{Q}W_{K}^{\top}x_{j}^{\top}$ . We first re-state the Theorem 3.1 below.
Theorem B.1. Let  $A = W_{V}W_{O}W_{Q}W_{K}^{\top}$  and let  $\lambda_{K}, \lambda_{Q}, \lambda_{V}, \lambda_{O}$  denote the largest singular values of  $W_{K}, W_{Q}, W_{V}, W_{O}$ , respectively. Consider the transformer in (5) with normalized inputs  $\| x_{t}\|_{2} = 1$  for all  $t$ . Let  $c, \epsilon > 0$  be constants. Assume that  $a_{t}x_{t + 1}^{\top} \geq (1 - \delta)\| a_{t}\|_{2}$  with  $\delta \leq \left(\frac{c\epsilon}{\lambda_Q\lambda_K\lambda_V\lambda_O}\right)^2$ . Then for all  $x_{\ell}$  satisfying  $x_{\ell}Ax_{\ell}^{\top} \geq c$  and  $x_{\ell}Ax_{\ell} \geq \epsilon^{-1}\max_{j\in [t], j \neq \ell}x_{j}Ax_{\ell}^{\top}$ , it holds that
$$
\frac {x _ {\ell} A x _ {\ell} ^ {\top}}{\| a _ {t} \| _ {2}} \left(\alpha_ {t, \ell} - 3 \epsilon\right) \leq x _ {t + 1} W _ {Q} W _ {K} ^ {\top} x _ {j} ^ {\top} \leq \frac {x _ {\ell} A x _ {\ell} ^ {\top}}{\| a _ {t} \| _ {2}} \left(\alpha_ {t, \ell} + 3 \epsilon\right) \tag {7}
$$
As a preparation of the proof, we first show two lemmas.
Lemma B.1. Let  $x_1, x_2 \in \mathbb{R}^{1 \times m}$  satisfies  $\| x_1 \|_2 = \| x_2 \|_2 = 1$  and  $x_1 x_2^\top \geq 1 - \delta$  for some  $\delta \in (0, 1)$ . Then for all  $y \in \mathbb{R}^{1 \times m}$  we have
$$
\left| x _ {1} y ^ {\top} - x _ {2} y ^ {\top} \right| \leq \sqrt {2 \delta} \left\| y \right\| _ {2}
$$
Proof. Let  $x_{2} = x_{2}^{\parallel} + x_{2}^{\perp}$  where
$$
x _ {2} ^ {\parallel} = x _ {1} x _ {2} ^ {\top} \cdot x _ {1}; \quad x _ {2} ^ {\perp} = x _ {2} - x _ {2} ^ {\parallel}
$$
Then it is easy to see that  $x_2^\perp x_1^\top = 0$ . By the Pythagorean Theorem, we have
$$
\left\| x _ {2} ^ {\perp} \right\| _ {2} ^ {2} = \left\| x _ {2} \right\| _ {2} ^ {2} - \left\| x _ {2} ^ {\parallel} \right\| _ {2} ^ {2} = \delta (2 - \delta)
$$
$$
\begin{array}{l} \left\| x _ {1} - x _ {2} \right\| _ {2} ^ {2} = \left\| \left(x _ {1} - x _ {2} ^ {\parallel}\right) - x _ {2} ^ {\perp} \right\| _ {2} ^ {2} \\ = \left\| \left(1 - x _ {1} x _ {2} ^ {\top}\right) x _ {1} - x _ {2} ^ {\perp} \right\| _ {2} ^ {2} \\ = \left(1 - x _ {1} x _ {2} ^ {\top}\right) ^ {2} + \left\| x _ {2} ^ {\perp} \right\| _ {2} ^ {2} \\ = 2 \delta \\ \end{array}
$$
Thus, the Cauchy-Schwarz inequality implies
$$
\left| x _ {1} y ^ {\top} - x _ {2} y ^ {\top} \right| \leq \left\| x _ {1} - x _ {2} \right\| _ {2} \cdot \left\| y \right\| _ {2} = \sqrt {2 \delta} \left\| y \right\| _ {2}
$$
Lemma B.2. Let  $\ell \in [t]$  be given. Suppose that  $x_{\ell}Ax_{\ell}^{\top} > \epsilon^{-1}\left|x_{j}Ax_{\ell}^{\top}\right|$  for all  $j\neq \ell$ . Then we have
$$
\left(\mathcal {S} (t) _ {\ell} - \epsilon\right) x _ {\ell} ^ {\top} a x _ {\ell} \leq x _ {\ell} ^ {\top} W _ {K} ^ {\top} W _ {Q} a _ {t} \leq \left(\mathcal {S} (t) _ {\ell} + \epsilon\right) x _ {\ell} ^ {\top} a x _ {\ell}
$$
$$
a _ {t} = \alpha_ {t} X _ {t - 1} W _ {V} W _ {O} = \left(\sum_ {j = 1} ^ {t - 1} \alpha_ {t, j} x _ {j}\right) W _ {V} W _ {O}
$$
$$
a _ {t} W _ {Q} W _ {K} ^ {\top} x _ {\ell} ^ {\top} = \left(\sum_ {j = 1} ^ {t - 1} \alpha_ {t, j} x _ {j}\right) W _ {V} W _ {O} W _ {Q} W _ {K} ^ {\top} x _ {\ell} ^ {\top} = \sum_ {j = 1} ^ {t - 1} \alpha_ {t, j} x _ {j} A x _ {\ell} ^ {\top}
$$
$$
\begin{array}{l} \left| a _ {t} W _ {Q} W _ {K} ^ {\top} x _ {\ell} ^ {\top} - \alpha_ {t, \ell} x _ {\ell} A x _ {\ell} ^ {\top} \right| = \left| \sum_ {j = 1, j \neq \ell} ^ {t - 1} \alpha_ {t, j} x _ {j} A x _ {\ell} ^ {\top} \right| \\ \leq \sum_ {j = 1, j \neq \ell} ^ {t - 1} \alpha_ {t, j} \left| x _ {j} A x _ {\ell} ^ {\top} \right| \\ \leq \epsilon x _ {\ell} A x _ {\ell} ^ {\top} \sum_ {j = 1, j \neq \ell} ^ {t - 1} \alpha_ {t, j} \\ \leq \epsilon x _ {\ell} A x _ {\ell} ^ {\top} \\ \end{array}
$$
where in the second inequality we use  $\epsilon^{-1}\left|x_jAx_\ell^\top\right| \leq x_\ell Ax_\ell^\top$  and in the third inequality we use  $\sum_{j=1, j \neq \ell}^{t-1} \alpha_{t,j} \leq \sum_{j=1}^{t-1} \alpha_{t,j} = 1$ . This implies that