SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for some function  $f: \mathbb{R}_{>0} \times \mathbb{R}_{>0} \to \mathbb{R}_{>0}$  with the property that, for any fixed  $C > 0$ , we have
$$
\lim  _ {c \rightarrow 0} f (C, c) = 0.
$$
We can thus pick a sufficiently small constant  $c > 0$ , depending only on  $C_{\gamma}$  and  $C$  such that  $f(C, c) < 1 - C_{\gamma}$  and this entire brute-force takes  $O(n^{2 - C_{\gamma}})$  time.
Case 2:  $t \geq c \log n$ .
Let  $a_1, \dots, a_n, b_1, \dots, b_n \in \{0, 1\}^d$  denote the input of Gap-ANN  $(n, d, t, \epsilon)$  (Definition 4.5), and recall from Remark 4.4 that we may assume each has half its entries 0 and half its entries 1. We will explain how to construct an Attention matrix using this instance.
Let  $C_0 \geq c$  be such that
$$
t := C _ {0} \log n. \tag {4}
$$
Let  $\beta > 0$  and  $\widetilde{d} \geq d$  denote parameters we will choose later (see Eq. (9) and Eq. (6), respectively). Define  $\tau > 0$  by
$$
\tau := \exp (\beta / 2). \tag {5}
$$
Intuitively, our goal in picking these parameters is that  $\tau$  will be an upper bound on entries of the attention matrix, i.e., we will have:
$$
\tau \geq \max  _ {i \in [ n ], j \in [ n ]} \exp (\beta \langle a _ {i}, b _ {j} \rangle / \widetilde {d}).
$$
We will make use of an algorithm for the AAttC  $(\widetilde{n},\widetilde{d},B,\epsilon_{a})$  problem, for the following parameters:
$$
\widetilde {n} := 2 n, \quad \widetilde {d} := 2 d, \tag {6}
$$
$$
B := C _ {b} \sqrt {\log n}, \quad \text {w h e r e} \quad C _ {b} := \sqrt {4 0 C / \left(C _ {0} \epsilon\right)}, \tag {7}
$$
$$
\epsilon_ {a} := n ^ {- C _ {a}}, \quad \text {w h e r e} \quad C _ {a} := 2 + C _ {b} ^ {2} \left(1 + C _ {0} / C\right). \tag {8}
$$
Furthermore, set
$$
\beta := B ^ {2}. \tag {9}
$$
We define  $Q \in \mathbb{R}^{\tilde{n} \times \tilde{d}}$  and  $K \in \mathbb{R}^{\tilde{n} \times \tilde{d}}$  as
$$
Q := \sqrt {\beta} \cdot \left[ \begin{array}{c c} a _ {1} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ a _ {2} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \vdots & \vdots \\ a _ {n} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \vdots & \vdots \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \end{array} \right] \quad \text {a n d} \quad K := \sqrt {\beta} \cdot \left[ \begin{array}{c c} b _ {1} ^ {\top} & \mathbf {0} _ {d} ^ {\top} \\ b _ {2} ^ {\top} & \mathbf {0} _ {d} ^ {\top} \\ \vdots & \vdots \\ b _ {n} ^ {\top} & \mathbf {0} _ {d} ^ {\top} \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} _ {d} ^ {\top} \\ \vdots & \vdots \\ \mathbf {0} _ {d} ^ {\top} & \mathbf {1} {_ d} ^ {\top} \end{array} \right].
$$
Since each entry of  $Q$  and  $K$  is either  $\sqrt{\beta}$  or 0, it follows that
$$
\| Q \| _ {\infty} \leq \sqrt {\beta} = B
$$
$$
\| K \| _ {\infty} \leq \sqrt {\beta} = B
$$
$$
\| Q K ^ {\top} / \widetilde {d} \| _ {\infty} \leq \frac {\beta \cdot \widetilde {d}}{\widetilde {d}} = \beta = B ^ {2}.
$$
Using the above construction of  $Q \in \mathbb{R}^{\widetilde{n} \times \widetilde{d}}$  and  $K \in \mathbb{R}^{\widetilde{n} \times \widetilde{d}}$ , we note that
$$
A := \exp (Q K ^ {\top} / \widetilde {d}) \in \mathbb {R} ^ {\widetilde {n} \times \widetilde {n}}
$$
$$
A = \left[ \begin{array}{c c c c c c c c} \exp (\beta \langle a _ {1}, b _ {1} \rangle / \widetilde {d}) & \exp (\beta \langle a _ {1}, b _ {2} \rangle / \widetilde {d}) & \dots & \exp (\beta \langle a _ {1}, b _ {n} \rangle / \widetilde {d}) & \tau & \tau & \dots & \tau \\ \exp (\beta \langle a _ {2}, b _ {1} \rangle / \widetilde {d}) & \exp (\beta \langle a _ {2}, b _ {2} \rangle / \widetilde {d}) & \dots & \exp (\beta \langle a _ {2}, b _ {n} \rangle / \widetilde {d}) & \tau & \tau & \dots & \tau \\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\ \exp (\beta \langle a _ {n}, b _ {1} \rangle / \widetilde {d}) & \exp (\beta \langle a _ {n}, b _ {2} \rangle / \widetilde {d}) & \dots & \exp (\beta \langle a _ {n}, b _ {n} \rangle / \widetilde {d}) & \tau & \tau & \dots & \tau \\ \exp (0) & \exp (0) & \dots & \exp (0) & \tau & \tau & \dots & \tau \\ \exp (0) & \exp (0) & \dots & \exp (0) & \tau & \tau & \dots & \tau \\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\ \exp (0) & \exp (0) & \dots & \exp (0) & \tau & \tau & \dots & \tau \end{array} \right].
$$
(Note that we do not explicitly compute all the entries of  $A$  in our algorithm; we will make use of it only through calling our algorithm for the Attention problem.)
For each  $(i,j)\in [n]\times [n]$  , we know that
$$
\begin{array}{l} A _ {i, j} = \exp \left(\beta \langle a _ {i}, b _ {j} \rangle / \tilde {d}\right) \\ \leq \exp (\beta \| a _ {i} \| _ {\infty} \cdot \| b _ {j} \| _ {\infty} \cdot d / \tilde {d}) \\ \leq \exp (\beta) \\ = \tau \tag {10} \\ \end{array}
$$
where the first step follows from definition of  $A \in \mathbb{R}^{\widetilde{n} \times \widetilde{n}}$ , the second step follows from  $\langle a_i, b_j \rangle \leq \|a_i\|_{\infty} \cdot \|b_j\|_{\infty} d$ , the third step follows from  $d < \widetilde{d}$  (see Eq. (6)), and the last step follows from definition of  $\tau$  (see Eq. (5)).
On the other hand, we know that for each  $(i,j)\in [n]\times [n]$
$$
A _ {i, j} \geq 0 \tag {11}
$$
since it is an exponential of an entry of  $QK^{\top} / \widetilde{d}$ .
Using Eq. (10) and Eq. (11), combined with our expression for  $A$ , it thus follows that
$$
n \tau \leq (A \mathbf {1} _ {\widetilde {n}}) _ {i} \leq 2 n \tau , \quad \forall i \in [ \widetilde {n} ].
$$
Since  $D_{i,i} = (A\mathbf{1}_{\tilde{n}})_i$  , thus we know that
$$
n \tau \leq D _ {i, i} \leq 2 n \tau , \quad \forall i \in [ \widetilde {n} ].
$$
Choose the vector  $v \in \mathbb{R}^{\tilde{n}}$  defined as
$$
v = \left[ \begin{array}{c} \mathbf {1} _ {n} \\ \mathbf {0} _ {n} \end{array} \right].
$$
We define  $\widetilde{t}$  as
$$
\widetilde {t} := \frac {1}{3} \exp (0. 2 5 \beta (1 - t / d)) / (2 n \tau). \tag {12}
$$
We can show that  $\widetilde{t} \geq \epsilon_{a}$  as follows:
$$
\begin{array}{l} \widetilde {t} = \frac {1}{6 n} \exp (0. 2 5 \beta (1 - t / d) - \beta) \\ = \frac {1}{6 n} \exp (- 0. 7 5 \beta - 0. 2 5 \beta t / d) \\ = \frac {1}{6 n} \exp \left(- 0. 7 5 \beta - 0. 2 5 \beta C _ {0} / C\right) \\ = \frac {1}{6} \exp \left(- 0. 7 5 \beta - 0. 2 5 \beta C _ {0} / C - \log n\right) \\ = \frac {1}{6} \exp \left(- 0. 7 5 C _ {b} ^ {2} \log n - 0. 2 5 C _ {b} ^ {2} \left(C _ {0} / C\right) \log n - \log n\right) \\ \geq n ^ {- C _ {a}} \\ \mathbf {\epsilon} = \epsilon_ {a}, \\ \end{array}
$$
where the second step follows from simple algebra, the third step follows from  $t = C_0 \log n$  (Eq. (4)) and  $d = C \log n$  (assumption in Lemma statement), the second step follows from choice of  $\beta$  (Eq. (7)), and the sixth step follows from choice of  $C_a$  (Eq. (8)), and the last step follows from Eq. (8).
Since  $\widetilde{t} \geq \epsilon_{a}$ , if we run an algorithm for Approximation Attention Computation (Definition 1.2) AAttC( $\widetilde{n}, \widetilde{d}, B, \epsilon_{a}$ ), where we pick  $V$  to be a matrix with one row  $v$  and the rest  $0$ , we can output a vector  $u \in \mathbb{R}^{\widetilde{n}}$  such that, for all  $i \in [\widetilde{n}]$ ,
$$
\left| u _ {i} - \left(D ^ {- 1} A v\right) _ {i} \right| <   \widetilde {t}.
$$
Note that using Remark 4.4, we have
$$
\left\| a _ {i} \right\| _ {2} ^ {2} / d = 0. 5, \quad \forall i \in [ n ],
$$
$$
\left\| b _ {j} \right\| _ {2} ^ {2} / d = 0. 5, \quad \forall j \in [ n ].
$$
Therefore, for any  $(i,j)\in [n]\times [n]$
$$
\begin{array}{l} \frac {1}{d} \langle a _ {i}, b _ {j} \rangle = \frac {1}{2 d} (\| a _ {i} \| _ {2} ^ {2} + \| b _ {j} \| _ {2} ^ {2} - \| a _ {i} - b _ {j} \| _ {2} ^ {2}) \\ = \frac {1}{2 d} (0. 5 d + 0. 5 d - \| a _ {i} - b _ {j} \| _ {2} ^ {2}) \\ = 0. 5 - 0. 5 \left\| a _ {i} - b _ {j} \right\| _ {2} ^ {2} / d, \\ \end{array}
$$
where the second step follows from  $\|a_i\|_2^2 = \|b_j\|_2^2 = d/2$ , and the last step follows from simple algebra.
Recall that our goal is to determine, for each  $i \in [n]$ , whether there is a  $j \in [n]$  such that  $\| a_i - b_j\|_2^2 \leq t$ , or whether  $\| a_i - b_j\|_2^2 \geq (1 + \epsilon_a)t$  for all  $j \in [n]$ . We will show next that we can distinguish these two cases by seeing whether  $u_i$  is greater than or less than the value  $\widetilde{t}_0 := 2\widetilde{t}$ .
Case 2a.
If there exists an  $(i,j)\in [n]\times [n]$  such that  $\| a_i - b_j\| _2^2\leq t$  , then
$$
\begin{array}{l} \beta \langle a _ {i}, b _ {j} \rangle / \widetilde {d} = 0. 5 \cdot \beta \langle a _ {i}, b _ {j} \rangle / d \\ \geq 0. 2 5 \cdot \beta (1 - t / d), \\ \end{array}
$$
where the first step follows from  $2d = \widetilde{d}$  (see Eq. (6)).
This means that
$$
\begin{array}{l} u _ {i} \geq \exp (0. 2 5 \beta (1 - t / d)) / (2 n \tau) - \widetilde {t} \\ = 3 \widetilde {t} - \widetilde {t} \\ = 2 \widetilde {t} \\ = \widetilde {t} _ {0}, \\ \end{array}
$$
where the second step follows from the definition of  $\widetilde{t}$  (see Eq. (12)), and the last step follows from the definition of  $\widetilde{t}_0$ .
Case 2b.
If for all  $(i,j)\in [n]\times [n]$ , we have  $\| a_{i} - b_{j}\|_{2}^{2} > t(1 + \epsilon)$ , this implies
$$
\beta \langle a _ {i}, b _ {j} \rangle / d \leq 0. 2 5 \beta \cdot (1 - t (1 + \epsilon) / d).
$$
Then, for all  $i\in [n]$
$$
\begin{array}{l} u _ {i} <   (n \cdot \exp (0. 2 5 \beta (1 - (1 + \epsilon) t / d))) / (n \tau) + \widetilde {t} \\ = \exp (0. 2 5 \beta (1 - t / d)) / (2 n \tau) \cdot (n / \exp (0. 2 5 \beta \epsilon t / d)) + \widetilde {t} \\ = 3 \widetilde {t} \cdot (n / \exp (0. 2 5 \beta \epsilon t / d)) + \widetilde {t} \\ \leq 3 \tilde {t} \cdot \frac {1}{4} + \tilde {t} \\ = 2 \widetilde {t} \\ = \tilde {t} _ {0}, \\ \end{array}
$$
where the third step follows from definition of  $\widetilde{t}$  (see Eq. (12)), the forth step follows from the calculation in Eq. (13) below, and the last step follows from  $\widetilde{t} = \widetilde{t}_0 / 2$ .
Finally, b our choice of  $\beta$  and  $t$ , we can see that
$$
\begin{array}{l} \exp (0. 2 5 \beta \epsilon t / d) = \exp ((0. 2 5 \beta \epsilon C _ {0} \log n) / d) \\ = \exp \left(0. 2 5 \beta \epsilon C _ {0} / C\right) \\ = \exp (1 0 \log n) \\ > 4 n, \tag {13} \\ \end{array}
$$
where the first step follows  $t = C_0 \log n$  (Eq. (4)), the second step follows from  $d = C \log n$ , and the third step follows from  $\beta = B^2$  (Eq. (9)) and the choice of  $B$  (Eq. (7)).
Recent Followup Work During the preparation of camera ready, we are aware of a number of new related work [KMZ23, HJK $^{+}$ 23, AS23, DLS23, GSWY23, DSZ23, GSY23] to this paper. This work studies softmax attention, and [KMZ23] shows to speed up the polynomial based attention unit via sketching techniques.  $\left[\mathrm{HJK}^{+}23\right]$  introduces hyperattention, which is able to handle long-context attention in near-linear time. This work is mainly focusing classical attention computation, [AS23] consider a more general tensor version computation of attention scheme. This work studies the inference computation, the recent work [DLS23, GSWY23] also considers how to speedup the training process which involves the computation of gradient and hessian. Inspired by hardness result of this work, [DSZ23] shows that there is a binary classification tasks (involves two datasets) that softmax attention can distinguish but linear attention cannot. [GSY23] provides a result for computing the attention matrix differentially privately. [GSYZ23] introduce a quantum algorithm for computing attention matrix.