SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Josh Alman*
Zhao Song†
In modern machine learning, inner product attention computation is a fundamental task for training large language models such as Transformer, GPT-1, BERT, GPT-2, GPT-3 and ChatGPT. Formally, in this problem, one is given as input three matrices  $Q, K, V \in [-B, B]^{n \times d}$ , and the goal is to construct the matrix  $\mathrm{Att}(Q, K, V) \coloneqq \mathrm{diag}(A\mathbf{1}_n)^{-1}AV \in \mathbb{R}^{n \times d}$ , where  $A = \exp(QK^\top / d)$  is the 'attention matrix', and  $\exp$  is applied entry-wise. Straightforward methods for this problem explicitly compute the  $n \times n$  attention matrix  $A$ , and hence require time  $\Omega(n^2)$  even when  $d = n^{o(1)}$  is small.
In this paper, we investigate whether faster algorithms are possible by implicitly making use of the matrix  $A$ . We present two results, showing that there is a sharp transition at  $B = \Theta(\sqrt{\log n})$ .
- If  $d = O(\log n)$  and  $B = o(\sqrt{\log n})$ , there is an  $n^{1 + o(1)}$  time algorithm to approximate  $\operatorname{Att}(Q, K, V)$  up to  $1 / \operatorname{poly}(n)$  additive error.
- If  $d = O(\log n)$  and  $B = \Theta(\sqrt{\log n})$ , assuming the Strong Exponential Time Hypothesis from fine-grained complexity theory, it is impossible to approximate  $\operatorname{Att}(Q, K, V)$  up to  $1 / \mathrm{poly}(n)$  additive error in truly subquadratic time  $n^{2 - \Omega(1)}$ .
This gives a theoretical explanation for the phenomenon observed in practice that attention computation is much more efficient when the input matrices have smaller entries.
Large language models (LLMs) such as Transformer  $\left[\mathrm{VSP}^{+}17\right]$ , BERT [DCLT18], GPT-3  $\left[\mathrm{BMR}^{+}20\right]$ , PaLM  $\left[\mathrm{CND}^{+}22\right]$ , and OPT  $\left[\mathrm{ZRG}^{+}22\right]$  can process natural language more effectively than smaller models or traditional algorithms. This means that they can understand and generate more complex and nuanced language, which can be useful for a variety of tasks such as language translation, question answering, and sentiment analysis. LLMs can also be adapted to multiple purposes without needing to be retained from scratch. Their power is particularly exemplified by the recent success of ChatGPT, a chat software by OpenAI built on top of GPT-3 [Ope22].
The key technical backbone of LLMs is the attention matrix [VSP+17, RNS+18, DCLT18, RWC+19, BMR+20]. An attention matrix is a square matrix whose rows and columns correspond to words or "tokens", and whose entries correspond to the correlations between these tokens in natural text. The attention matrix is then used to calculate the importance of each input token in a sequence when producing an output. In an attention mechanism, each input token is given a weight or score, which reflects its importance or relevance to the current output being generated. These scores are calculated based on a comparison between the current output state and the input states, using a similarity function.
$\dagger$  zsong@adobe.com. Adobe Research.
More formally, the attention matrix is defined as follows. Let  $Q \in \mathbb{R}^{n \times d}$  be the matrix of query tokens, and  $K \in \mathbb{R}^{n \times d}$  be the matrix of key tokens. (We focus here on the case when  $d = n^{o(1)}$ , so  $d \ll n$ .) The attention matrix is an  $n \times n$  matrix  $A$  where the rows and columns correspond to the input tokens in the sequence. Each entry in the matrix represents the attention weight or score between a particular input token (query token  $Q$ ) and a particular output token (key token  $K$ ). The diagonal entries of the matrix represent self-attention scores, which measure the importance of each token with respect to itself.
The major bottleneck to speeding up LLM operations (in the case of modeling long sequences with large  $n$ ) is the time to perform attention matrix computations [VSP+17, RNS+18, DCLT18, RWC+19, BMR+20, WLK+20, KKL20]. These computations ask us to multiply the attention matrix  $A$  with another value token matrix  $V \in \mathbb{R}^{n \times d}$ .
We formally define Attention computation as follows. Throughout this paper, we write exp to denote the entry-wise exponential for matrices.
Definition 1.1 (Exact Attention Computation EAttC(n, d)). Given three matrices  $Q$ ,  $K$ ,  $V \in \mathbb{R}^{n \times d}$ , output the  $n \times d$  matrix Att( $Q$ ,  $K$ ,  $V$ ) defined by
$$
\operatorname {A t t} (Q, K, V) := D ^ {- 1} A V
$$
where  $A\in \mathbb{R}^{n\times n}$  and diagonal matrix  $D\in \mathbb{R}^{n\times n}$  are defined as
$$
A := \exp (Q K ^ {\top} / d), \quad a n d \quad D := \operatorname {d i a g} \left(A \mathbf {1} _ {n}\right).
$$
The straightforward algorithm for this problem computes the matrix  $A$  and then performs the multiplications  $D^{-1}AV$ , in time  $n^{2 + o(1)}$ . Since  $A$  is an  $n \times n$  matrix with  $n^2$  entries, it is impossible to improve on this much while explicitly computing the matrix  $A$ . However, the input to the problem is not  $A$ , but rather the three matrices  $Q, K, V$  which each have only  $n^{1 + o(1)}$  entries. An algorithm which only implicitly makes use of  $A$ , without explicitly computing all its entries, could hope to run in almost linear time!
In this paper, we investigate the possibility of accelerating attention computations in this way. The two main questions we address are:
- Q1. When can we perform attention computations in almost linear time  $n^{1 + o(1)}$ ?
- Q2. When can we prove that subquadratic-time algorithms for attention computations are impossible?
In most LLMs, it suffices to approximately perform attention computations throughout the inference process as long as there are reasonable precision guarantees [CGRS19, KKL20,  $\mathrm{WLK}^{+}20$ , DKOD20, KVPF20,  $\mathrm{CDW}^{+}21$ ,  $\mathrm{CDL}^{+}22$ ,  $\mathrm{LWD}^{+}23$ ,  $\mathrm{ZSZ}^{+}23$ ]. We therefore focus here on approximate attention computation, which can potentially be performed even faster than exact computation. Mathematically, we define the approximate version of EAttC as follows.
Definition 1.2 (Approximate Attention Computation  $\mathsf{AAttC}(n,d,B,\epsilon_a)$ ). Let  $\epsilon_{a} > 0$  and  $B > 0$  be parameters. Given three matrices  $Q,K,V\in \mathbb{R}^{n\times d}$ , with the guarantees that  $\| Q\|_{\infty}\leq B$ ,  $\| K\|_{\infty}\leq B$ , and  $\| V\|_{\infty}\leq B$ , output a matrix  $T\in \mathbb{R}^{n\times d}$  which is approximately equal to  $D^{-1}AV$ , meaning,
$$
\| T - D ^ {- 1} A V \| _ {\infty} \leq \epsilon_ {a}.
$$
Here, for a matrix  $M \in \mathbb{R}^{n \times n}$ , we write  $\| M \|_{\infty} \coloneqq \max_{i,j} |M_{i,j}|$ .
Again, the straightforward algorithm for this problem runs in time  $O(n^{2}d) \leq n^{2 + o(1)}$ , but the input size is only  $O(nd) \leq n^{1 + o(1)}$ . Our goal is to investigate when faster algorithms are possible in terms of the parameters  $d, B$ , and  $\epsilon_{a}$ .
We focus on the natural setting where  $d = O(\log n)$  (the setting where we model long sequences) and  $\epsilon_{a} = 1 / \mathrm{poly}(n)$  (low enough error so that attention computations over an entire network can be combined). Our main results show that whether or not there is a fast algorithm for AAattC critically depends on  $B$ , the magnitudes of the entries in the input matrices.
We first show a lower bound, that when  $B \geq \Omega(\sqrt{\log n})$ , it is impossible to design a truly subquadratic-time algorithm. Our lower bound makes use of the Strong Exponential Time Hypothesis (SETH) [IP01], a popular conjecture [Wil18] from the area of fine-grained complexity regarding the time required to solve  $k$ -SAT. (See Section 4 below where we discuss SETH in more detail.)
Theorem 1.3 (Lower bound, informal version of Theorem 4.6). Assuming SETH, for every  $q > 0$ , there are constants  $C, C_a, C_b > 0$  such that: there is no  $O(n^{2 - q})$  time algorithm for the problem AAttC( $n, d = C \log n, B = C_b \sqrt{\log n}, \epsilon_a = n^{-C_a}$ ).
Our second complementary result is a new algorithm, showing that when  $B < o(\sqrt{\log n})$ , the problem can be solved very efficiently, in almost linear time.
Theorem 1.4 (Upper bound, informal version of Theorem 3.8). There is an algorithm (Algorithm 1) that solves  $\mathsf{AAttC}(n,d = O(\log n),B = o(\sqrt{\log n}),\epsilon_a = 1 / \mathrm{poly}(n))$  in time  $n^{1 + o(1)}$ .
Our Theorems 1.3 and 1.4 show that the attention computation problem AAttC exhibits a very tight transition at  $B = \Theta(\sqrt{\log n})$  from almost linear time to trivial quadratic time. When  $B < o(\sqrt{\log n})$  is smaller, the problem can be solved in almost linear time  $n^{1 + o(1)}$  in the input size, using our algorithm for Theorem 1.4. When  $B \geq \Omega(\sqrt{\log n})$  is greater, our algorithm from Theorem 1.4 no longer applies, and furthermore our lower bound from Theorem 1.3 shows that it is impossible to solve the problem in truly subquadratic time, no matter what algorithmic techniques one uses (assuming SETH).
It has been observed in LLM implementations in practice that computations are much faster when one assumes that the matrix entries are bounded or can be well-approximated using a small number of bits (see, e.g., [ZBIW19, Section 2] and [KVPF20, Section 3.2.1]). Our work can be viewed as giving a theoretical explanation for this phenomenon, and helping to explain why techniques like quantization [ZBIW19] and low-degree polynomial approximation [KVPF20] have been so effective in practice.
A recent work by Zandieh, Han, Daliri, and Karbasi [ZHDK23] was the first to give an algorithm with provable guarantees for attention approximation. Their algorithm makes use of locality sensitive hashing (LSH) techniques [CKNS20] which, as we will discuss next, is quite different from our algorithm for Theorem 1.4 which uses the polynomial method [ACSS20, AA22].
In the case when  $d = o(\log^2 n)$ , they achieve a running time of roughly  $O(n^{1.17} \cdot d / \epsilon_r^2)$ , where  $\epsilon_r$  is a relative error parameter (which is similar, though not exactly the same, as our  $\epsilon_a$  from Definition 1.2). In particular, their algorithm applies for larger  $d$  than ours (we require  $d = O(\log n)$ ), but we achieve almost linear time  $n^{1 + o(1)}$  (whereas their running time is bounded below by  $\Omega(n^{1.17})$ ), and our algorithm can handle any polynomial error  $\epsilon_a = 1 / \mathrm{poly}(n)$  (whereas they require  $\epsilon_r \geq 1 / n^{o(1)}$  to not increase the running time by a polynomial factor).
It is natural to wonder whether further improvements are possible by combining our techniques with those of [ZHDK23]. However, our lower bound of Theorem 1.3 shows that our algorithm of Theorem 1.4 is already essentially tight and cannot be substantially improved.
Another recent work by Keles, Wijewardena, and Hedge [KWH23] was the first to prove a lower bound for attention computation assuming SETH. They prove, among other results, that AAttC cannot be solved in truly subquadratic time in the case when  $d = \omega (\log n)$ . Our Theorem 1.3 improves their result to also hold for  $d = \Theta (\log n)$ , and to show how the complexity changes with the magnitude of entries  $B$  (which is not studied by [KWH23]). As we discuss more shortly, both our lower bound proof and [KWH23] use the high-level technique of [BIS17], although our more fine-grained analysis of the parameters  $d, B$  requires a more intricate analysis and the use of other techniques from fine-grained complexity related to approximate nearest neighbor search [Rub18] and the polynomial method [AA22].
1.2 Technique Overview
Our high-level approach is to make use of similarities between attention computation and other computational problems related to Kernel Density Estimation (KDE). Such a relationship was
investigated by recent work  $\left[\mathrm{TBY}^{+}19, \mathrm{ZHDK}23\right]$ . In particular, [ZHDK23] was inspired to apply LSH techniques to attention computation because of the prevalence of LSH in KDE algorithms [CS17, BCIS18, CS19, CKNS20]. The main conceptual idea behind our results is that different techniques from the KDE literature, other than LSH, can be modified to apply in this setting and yield tight algorithms and lower bounds.
To design our algorithm for Theorem 1.3, we instead build off of a different line of work on KDE which makes use of the 'polynomial method in algorithm design'. Suppose  $M \in \mathbb{R}^{n \times n}$  is a matrix,  $f: \mathbb{R} \to \mathbb{R}$  is a function, and let  $f(M)$  denote the matrix one gets by applying  $f$  entry-wise to  $M$ . The polynomial method is a technique for finding low-rank approximations of  $f(M)$ . It shows that if  $M$  has low rank, and if  $f$  can be approximated by a low-degree polynomial, then the matrix  $f(M)$  is very close to a low-rank matrix whose low-rank decomposition can be computed efficiently.
To use this to solve AAttC, we make use of a recent result which bounds the degree required to approximate the exponential function by a polynomial [AA22] in order to find a low-rank approximation of the attention matrix  $A$ . Prior work [ACSS20,  $\mathrm{ACM}^{+}20$ , AA22] applied these polynomials in a similar way to solve the Gaussian KDE problem; our main observation is that by an appropriate rescaling, this approach can be modified to apply to AAttC as well.
The proof of our lower bound Theorem 1.3 builds off of another line of work on the fine-grained complexity of KDE problems [BIS17, ACSS20, AA22]. The main idea is to give a fine-grained reduction from the well-studied problem of Approximate Nearest Neighbor search ANN. In ANN, one is given as input  $n$  vectors of dimension  $d$ , and an error parameter  $\epsilon > 0$ , and the goal is to find a pair of vectors whose distance is at most  $(1 + \epsilon)$  times the minimum distance between any pair of the vectors. The straightforward algorithm for ANN runs in quadratic time, and it is known that it is impossible to solve ANN in truly subquadratic time assuming SETH [Rub18].
In order to prove our lower bound, we show that AAttC can be used to solve ANN. The key idea is that, if the matrices  $Q$  and  $K$  from AAttC are formed by concatenating the input vectors to the ANN problem, then the nearest neighbor vectors correspond to the largest entries of the attention matrix  $A$ . It is not immediately clear that AAttC can be used to detect large entries of  $A$ , since the output is rescaled by the matrix  $D^{-1}$ , but we show that this can be overcome with some modifications to the input vectors which approximately balance the rows of  $A$ . Prior work [BIS17, ACSS20, AA22] used a very similar approach to give lower bounds for KDE problems, although KDE doesn't involve any rescaling factors.
In Section 2, we introduce relevant notation and tools from prior work. In Section 3, we present and analyze our attention algorithm. In Section 4, we prove our fine-grained attention lower bound. In Section 5, we provide a conclusion for this paper.
We work in the standard real-RAM model and assume arithmetic operations on real numbers can be performed in constant time in our algorithms.
We use  $\mathcal{T}_{\mathrm{mat}}(a,b,c)$  to denote the time to multiply an  $a\times b$  matrix with another  $b\times c$  matrix. In fact, we will only make use of the straightforward, practical bound  $\mathcal{T}_{\mathrm{mat}}(a,b,c)\leq O(abc)$ . In principle, fast theoretical matrix multiplication algorithms could be used instead to improve this bound and speed up our algorithms here (in exchange for making them less practical). That said, because of our parameter settings<sup>3</sup>, we will see that faster matrix multiplication could only improve low-order terms in our running times.
For any positive integer, we use  $[n]$  to denote set  $\{1,2,\dots ,n\}$ .
For a matrix  $M$ , we write  $\| M \|_{\infty}$  to denote its  $\ell_{\infty}$  norm, i.e.,  $\| M \|_{\infty} := \max_{i,j} |M_{i,j}|$ . For a matrix  $M$ , we use  $M^{\top}$  to denote its transpose.
We use  $\mathbf{1}_n$  to denote a length- $n$  vector whose entries are all 1s. We use  $\mathbf{0}_n$  to denote a length- $n$  vector whose entries are all 0s.
<sup>3</sup>We will make use of  $\mathcal{T}_{\mathrm{mat}}(n, n^{o(1)}, n^{o(1)})$ , which can be solved straightforwardly in time  $n^{1 + o(1)}$ , and which cannot be solved much faster since it has input size  $n^{1 + o(1)}$ .
For any matrix  $A \in \mathbb{R}^{n \times n}$ , we use  $\exp(A) \in \mathbb{R}^{n \times n}$  to denote the matrix where  $\exp(A)_{i,j} = \exp(A_{i,j})$ . In other words, all the  $\exp()$  operators in this paper are applied entry-wise to matrices. In particular, we will not use matrix exponentials in this paper.
For a vector  $x \in \mathbb{R}^n$ , we use  $\|x\|_0$  to denote its number of non-zero entries, we use  $\|x\|_1$  to denote its  $\ell_1$  norm, i.e.,  $\|x\|_1 := \sum_{i=1}^{n} |x_i|$ , and we use  $\|x\|_2$  to denote its  $\ell_2$  norm, i.e.,  $\|x\|_2 := (\sum_{i=1}^{n} |x_i|^2)^{1/2}$ . For a vector  $x$ , we use  $x^\top$  to denote its transpose.
2.1 Additive Error for Polynomial Approximation
Our algorithm for attention computation will critically make use of a polynomial approximation for the exponential function. In particular, we use the following tight construction from previous work [AA22].
Lemma 2.1 ([AA22]). Let  $B > 1$  and let  $\epsilon \in (0,0.1)$ . There is a polynomial  $P:\mathbb{R}\to \mathbb{R}$  of degree  $g\coloneqq \Theta \left(\max \left\{\frac{\log(1 / \epsilon)}{\log(\log(1 / \epsilon) / B)},B\right\}\right)$  such that for all  $x\in [0,B]$ , we have
$$
| P (x) - \exp (x) | <   \epsilon .
$$
Moreover,  $P$  can be computed efficiently: its coefficients are rational numbers with  $\mathrm{poly}(g)$ -bit integer numerators and denominators which can be computed in  $\mathrm{poly}(g)$  time.
2.2 From Additive Error to Relative Error
We note that in our setting, Lemma 2.1 can be used to give a relative error approximation as well:
Corollary 2.2. Let  $B > 1$  and let  $\epsilon \in (0,0.1)$ . There is a polynomial  $P: \mathbb{R} \to \mathbb{R}$  of degree  $g := \Theta\left(\max\left\{\frac{\log(1 / \epsilon)}{\log(\log(1 / \epsilon) / B)}, B\right\}\right)$  such that for all  $x \in [-B, B]$ , we have
$$
| P (x) - \exp (x) | <   \epsilon \cdot \exp (x).
$$
Proof. By Lemma 2.1, there is a polynomial  $Q: \mathbb{R} \to \mathbb{R}$  of degree  $g = \Theta(\{\frac{\log(1 / \epsilon)}{\log(\log(1 / \epsilon) / B)}, B\})$  such that, for all  $y \in [0, 2B]$  we have  $|Q(y) - \exp(y)| \leq \epsilon$ . Our desired polynomial is the rescaled  $P(x) := Q(x + B) / \exp(B)$ . Indeed, for any  $x \in [-B, B]$ , we have  $\exp(x) \geq \exp(-B)$ , and so
$$
\begin{array}{l} | P (x) - \exp (x) | = | Q (x + B) / \exp (B) - \exp (x) | \\ = | Q (x + B) - \exp (x + B) | / \exp (B) \\ \leq \epsilon / \exp (B) \\ \leq \epsilon \cdot \exp (x), \\ \end{array}
$$
3 Attention Algorithm
In this section, we show how to use polynomial approximations for the exponential function in order to approximately perform attention computations. In Section 3.1, we define the type of low-rank matrix approximation which we will use. In Section 3.2, we show how polynomial approximations can give rise to such low-rank matrix approximations. In Section 3.3, we bound the entries of the matrix  $QK^{\top} \in \mathbb{R}^{n \times n}$  (before converting it to the attention matrix) to confirm that our polynomial approximation applies. In Section 3.4, we state our main technique for approximating the attention matrix. In Section 3.5, we show how to control the error propagation from  $A$  to the rescaling matrix  $D$ . In Section 3.6, we further explain how to control the error propagation from  $D$  and  $A$  to the resulting attention matrix. Finally, in Section 3.7, we conclude our general algorithm, and in Section 3.8, we appropriately select the parameters to achieve almost linear time.
3.1 Matrix Low-Rank Approximation
Definition 3.1. Let  $r \geq 1$  denote a positive integer. Let  $\epsilon \in (0, 0.1)$  denote an accuracy parameter.
Given a matrix  $A \in \mathbb{R}_{\geq 0}^{n \times n}$ , we say  $\widetilde{A} \in \mathbb{R}_{\geq 0}^{n \times n}$  is an  $(\epsilon, r)$ -approximation of  $A$  if
-  $\widetilde{A} = U_1 \cdot U_2^\top$  for some matrices  $U_1, U_2 \in \mathbb{R}^{n \times r}$  (i.e.,  $\widetilde{A}$  has rank at most  $r$ ), and
-  $|\widetilde{A}_{i,j} - A_{i,j}| \leq \epsilon \cdot A_{i,j}$  for all  $(i,j) \in [n]^2$ .
3.2 From Low Degree Polynomials to Low Rank Matrices
Lemma 3.2. Let  $M = XY^{\top} \in \mathbb{R}^{n \times n}$  denote a matrix with  $X, Y \in \mathbb{R}^{n \times d}$ . Let  $P(x)$  denote a degree- $g$  polynomial, and define  $r = \binom{2(g+d)}{2g}$ .
There is an algorithm that runs in  $O(nrg)$  time and, given as input the matrix  $X, Y$ , constructs matrices  $U_1, U_2 \in \mathbb{R}^{n \times r}$  such that  $P(M) = U_1U_2^\top$ . (Here,  $P(M)$  denotes the entry-wise application of  $P$  to  $M$ .)
Due to space limitation, we defer the proof of Lemma 3.2 to Appendix A.
3.3 Matrix  $QK^{\top}$  Has Bounded Entries
Lemma 3.3 (Bounded entry). Suppose  $B \geq 1$  and matrices  $Q, K \in \mathbb{R}^{n \times d}$  have  $\| Q \|_{\infty} \leq B$  and  $\| K \|_{\infty} \leq B$ . Then, we have
$$
\| Q K ^ {\top} / d \| _ {\infty} \leq B ^ {2}.
$$
Proof. For each  $(i,j)\in [n]\times [n]$ , we have
$$
\begin{array}{l} | (Q K ^ {\top}) _ {i, j} | = \left| \sum_ {l = 1} ^ {d} Q _ {i, l} K _ {j, l} \right| \\ \leq d \cdot \| Q \| _ {\infty} \cdot \| K \| _ {\infty} \\ \leq d \cdot B ^ {2}, \\ \end{array}
$$
3.4 Key Lemma
Our key lemma shows that, even though the attention matrix  $A$  may have full rank, it has a low-rank approximation that is easy to compute:
Lemma 3.4. Suppose  $Q, K \in \mathbb{R}^{n \times d}$ , with  $\|Q\|_{\infty} \leq B$ , and  $\|K\|_{\infty} \leq B$ . Let  $A := \exp(QK^{\top} / d) \in \mathbb{R}^{n \times n}$ . For accuracy parameter  $\epsilon \in (0,1)$ , there is a positive integer  $g$  bounded above by
$$
g = O \Big (\max  \left\{\frac {\log (1 / \epsilon)}{\log (\log (1 / \epsilon) / B ^ {2})}, B ^ {2} \right\} \Big),
$$
and a positive integer  $r$  bounded above by
$$
r \leq \left( \begin{array}{c} 2 (g + d) \\ 2 g \end{array} \right)
$$
such that: There is a matrix  $\widetilde{A} \in \mathbb{R}^{n \times n}$  that is an  $(\epsilon, r)$ -approximation (Definition 3.1) of  $A \in \mathbb{R}^{n \times n}$ . Furthermore, the matrices  $U_{1}$  and  $U_{2}$  defining  $\widetilde{A}$  can be computed in  $O(n \cdot r)$  time.
Proof. Let  $M \coloneqq QK^{\top} / d$ . From Lemma 3.3, we know that  $\| M \|_{\infty} \leq B^{2}$ . Thus, applying Corollary 2.2 (with bound  $B^{2}$  on its entries), there is a degree- $g$  polynomial  $P$  such that the matrix  $\widetilde{A} = P(M)$  is an  $(\epsilon, r)$ -approximation to  $A$  (See the definition of  $(\epsilon, r)$ -approximation in Definition 3.1.) We can then compute  $U_{1}, U_{2}$  using Lemma 3.2, which gives the bound
$$
r \leq \left( \begin{array}{c} 2 (g + d) \\ 2 g \end{array} \right).
$$
3.5 From  $A$  to  $D$
Lemma 3.5. Let  $A \in \mathbb{R}^{n \times n}$  be any matrix whose entries are all positive and  $\epsilon_A \in (0, 0.1)$  be any parameter. Let  $\widetilde{A} \in \mathbb{R}^{n \times n}$  be any matrix such that, for all  $(i, j) \in [n] \times [n]$ , we have
$$
\left| \widetilde {A} _ {i, j} - A _ {i, j} \right| \leq \epsilon_ {A} \cdot A _ {i, j}.
$$
Define the matrices  $D, \widetilde{D} \in \mathbb{R}^{n \times n}$  by  $D = \mathrm{diag}(A\mathbf{1}_n)$  and  $\widetilde{D} = \mathrm{diag}(\widetilde{A}\mathbf{1}_n)$ . Then, for all  $i \in [n]$  we have
$$
\left| \widetilde {D} _ {i, i} - D _ {i, i} \right| \leq \epsilon_ {A} \cdot D _ {i, i}.
$$
Due to space limitation, we defer the proof of Lemma 3.5 into Appendix A.
3.6 From  $A$  and  $D$  to Attention Matrix