SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 3.6. Let  $\epsilon_A, \epsilon_D \in (0, 0.1)$  and  $B > 1$  be parameters, and let  $V \in \mathbb{R}^{n \times d}$  denote a matrix with  $\| V \|_{\infty} \leq B$ . Let  $A \in \mathbb{R}^{n \times n}$  be any matrix whose entries are all positive, and let  $\widetilde{A} \in \mathbb{R}^{n \times n}$  be a matrix such that, for all  $(i, j) \in [n] \times [n]$  we have
Let  $D, \widetilde{D} \in \mathbb{R}^{n \times n}$  be any diagonal matrices with positive entries on their diagonals, with the property that, for all  $i \in [n]$ , we have
$$
\left| \widetilde {D} _ {i, i} - D _ {i, i} \right| \leq \epsilon_ {D} \cdot D _ {i, i}.
$$
$$
\| \widetilde {D} ^ {- 1} \widetilde {A} V - D ^ {- 1} A V \| _ {\infty} \leq (\epsilon_ {A} + \epsilon_ {D}) \cdot B.
$$
Due to space limitation, we delay the proof of Lemma 3.6 to Appendix A.
3.7 Main Upper Bound
Theorem 3.7. For positive integers  $n, d$  and real parameters  $\epsilon > 0$  and  $B > 1$ , there are positive integers  $g = \Theta(\max\{\frac{\log(1/\epsilon)}{\log(\log(1/\epsilon)/B^2)}, B^2\})$  and  $r = \binom{2(g+d)}{2d}$  such that: There is an algorithm (Algorithm 1) that runs in  $O(\mathcal{T}_{\mathrm{mat}}(n, r, d) + nrg)$  time to solve AAttC( $n, d, B, \epsilon$ ) (Definition 1.2).
Proof. The running time of each step is shown in Algorithm 1; its running time follows from Lemma 3.4. Its correctness follows from Lemma 3.5 and Lemma 3.6.  $\square$
Algorithm 1 Our Algorithm
1: procedure POLYATTENTION  $(Q\in \mathbb{R}^{n\times d},K\in \mathbb{R}^{n\times d},V\in \mathbb{R}^{n\times d},n,d,B,\epsilon)$  ▷ Theorem 1.4  
2: ▷  $\epsilon$  is the accuracy output  
3: ▷  $\| Q\|_{\infty}$ ,  $\| K\|_{\infty}$ ,  $\| V\|_{\infty}$  ≤ B  
4:  $g\gets O(\max \{\frac{\log(1 / \epsilon)}{\log(\log(1 / \epsilon) / B^2)},B^2\})$   
5:  $r\gets \binom{2(g+d)}{2d}$   
6: Construct  $U_1,U_2\in \mathbb{R}^{n\times r}$  via Lemma 3.4 ▷  $O(nrg)$  time  
7:  $\widetilde{w}\gets U_1\cdot (U_2^\top \mathbf{1}_n)$  ▷  $O(nr)$  time  
8:  $\widetilde{D}^{-1} = \mathrm{diag}(\widetilde{w}^{-1})$  ▷  $O(n)$  time  
9: Compute  $U_2^\top V\in \mathbb{R}^{r\times d}$  ▷ Takes  $\mathcal{T}_{\mathrm{mat}}(r,n,d)$  time  
10: Compute  $U_1\cdot (U_2^\top V)$  ▷  $\mathcal{T}_{\mathrm{mat}}(n,r,d)$  time  
11:  $T\gets \widetilde{D}^{-1}\cdot (U_1\cdot (U_2^\top V))$  ▷  $O(nd)$  time  
12: return  $T$  ▷  $T\in \mathbb{R}^{n\times d}$   
13: end procedure
3.8 Proof of Theorem 1.4
Theorem 3.8 (Upper bound, formal statement of Theorem 1.4).  $\mathsf{AAttC}(n,d = O(\log n),B = o(\sqrt{\log n}),\epsilon_{a} = 1 / \mathrm{poly}(n))$  can be solved in time  $\mathcal{T}_{\mathrm{mat}}(n,n^{o(1)},d) = n^{1 + o(1)}$ .
Proof. If we select the parameters
$$
B = o (\sqrt {\log n}), \quad \epsilon = 1 / \operatorname {p o l y} (n), \quad d = O (\log n)
$$
in Theorem 3.7, then we see that
$$
\begin{array}{l} g = O (\max \{\frac {\log (1 / \epsilon)}{\log (\log (1 / \epsilon) / B ^ {2})}, B ^ {2} \}) \\ = O \left(\max  \left\{\frac {\log (n)}{\log \left(\log (n) / B ^ {2}\right)}, B ^ {2} \right\}\right) \\ = O \left(\max  \left\{\frac {\log n}{\log \log n}, o (\log n) \right\}\right) \\ = o (\log n), \\ \end{array}
$$
where the second step follows from  $\epsilon = 1 / \mathrm{poly}(n)$  and the third step follows from  $B = o(\sqrt{\log n})$ .
Since  $g = o(\log n)$ , let us write  $g = (\log n) / f$  for some  $f = \omega(1)$ . We thus have that
$$
r = \left( \begin{array}{c} 2 (d + g) \\ 2 g \end{array} \right) \leq \left(\frac {e (d + g)}{g}\right) ^ {2 g} \leq 2 ^ {O (g \log ((\log n) / g))} \leq 2 ^ {O (\log n \log (f) / f)} <   2 ^ {o (\log n)} <   n ^ {o (1)}.
$$
The second step follows from the generic bound  $\binom{a}{b} \leq (ea/b)^{b}$  for  $1 \leq b \leq a$ , and the third step uses that  $d = O(\log n)$ .
Since  $d, r, g$  are all bounded by  $n^{o(1)}$ , our final running time is  $n^{1 + o(1)}$  as desired.
4 Hardness
In this section, we prove our fine-grained lower bound for attention computation. In Section 4.1, we state the Strong Exponential Time Hypothesis (SETH), the main hardness assumption we will use. In Section 4.2, we define the approximate nearest neighbor search problem, and its known hardness assuming SETH. Finally, in Section 4.3, we give a reduction from approximate nearest neighbor search to attention computation, which implies our hardness result.
4.1 Fine-Grained Hypotheses
The Strong Exponential Time Hypothesis (SETH) was introduced by Impagliazzo and Paturi [IP01] over 20 years ago. It is a strengthening of the  $\mathsf{P} \neq \mathsf{NP}$  conjecture, which asserts that our current best SAT algorithms are roughly optimal:
Hypothesis 4.1 (Strong Exponential Time Hypothesis (SETH)). For every  $\epsilon >0$  there is a positive integer  $k\geq 3$  such that  $k$  -SAT on formulas with  $n$  variables cannot be solved in  $O(2^{(1 - \epsilon)n})$  time, even by a randomized algorithm.
SETH is a popular conjecture which has been used to prove fine-grained lower bounds for a wide variety algorithmic problems. See, for instance, the survey [Wi18].
4.2 Nearest Neighbor Search
We will make use of a known relationship between SETH and approximate nearest neighbor search.
Definition 4.2 (Approximate Hamming Nearest Neighbor Search (ANN)). For a parameter  $\epsilon >0$  in the  $(1 + \epsilon)$ -Approximate Hamming Nearest Neighbor Search problem for  $n$  vectors of dimension  $d$ , we are given as input two sets  $A,B\subset \{0,1\} ^d$  with  $|A| = |B| = n$ , and our goal is to find an  $a^* \in A$  and  $b^{*}\in B$  satisfying  $\| a^{*} - b^{*}\|_{0}\leq (1 + \epsilon)\cdot \min_{a\in A,b\in B}\| a - b\|_{0}$ .
(This is sometimes called the 'bichromatic' ANN problem, and a monochromatic version has also been studied; see, for instance, [SM19].) Rubinstein [Rub18] showed that for certain parameters, it is impossible to substantially improve on the straightforward quadratic-time algorithm for ANN assuming SETH:
Lemma 4.3 ([Rub18]). Assuming SETH, for every  $q > 0$ , there are  $\epsilon \in (0,1)$  and  $C > 0$  such that  $(1 + \epsilon)$ -Approximate Hamming Nearest Neighbor Search in dimension  $d = C$  log  $n$  requires  $\Omega(n^{2 - q})$  time.
Remark 4.4. We may assume that 4.3 holds even in the special case where each input vector from  $A$  and  $B$  has half its entries equal to 0 and half equal to 1. Indeed, for any vector  $a \in \{0,1\}^d$ , we can construct a new vector  $\widetilde{a} \in \{0,1\}^{2d}$  given by  $\widetilde{a} = \left[a^\top \overline{a}^\top\right]^\top$ . Here  $\overline{a} \in \{0,1\}^d$  is the binary complement of vector  $a$ , i.e.,  $\overline{a}_i = 1 - a_i$  for all  $i \in [d]$ . Thus,  $\|\widetilde{a}\|_0 = d$ . We can similarly construct a new vector  $\widetilde{b} \in \{0,1\}^{2d}$  for each  $b \in B$ . After this transformation, for any  $a \in A$  and  $b \in B$ , we have  $\|\widetilde{a} - \widetilde{b}\|_0 = 2 \cdot \|a - b\|_0$ , so it suffices to find an approximate nearest neighbor among these transformed vectors.
For convenience in our the analysis, we define a gap version of approximate nearest neighbor search problem  $\mathrm{Gap - ANN}(n,d,t,\epsilon)$ .
Definition 4.5 (Gap approximate nearest neighbor search (Gap-ANN(n, d, t, ε))). Let  $n, d$  denote two positive integers. Let  $t > 0$  denote a threshold parameter. Let  $\epsilon$  denote a accuracy parameter. Given two sets of points  $A = \{a_1, \dots, a_n\} \subset \{0, 1\}^d$  and  $B = \{b_1, \dots, b_n\} \subset \{0, 1\}^d$ : For each  $i \in [n]$ , we need to distinguish the following two cases
- Case 1. There exists a  $j \in [n]$  such that  $\| a_i - b_j \|_0 < t$ .
Case 2. For all  $j \in [n]$  we have  $\| a_{i} - b_{j}\|_{2}^{2} \geq (1 + \epsilon)\cdot t$ .
An algorithm for Gap-  $\mathsf{ANN}(n,d,t,\epsilon)$  can be called  $\log (nd)$  times to binary search for the answer to ANN, so Lemma 4.3 holds as well for Gap-  $\mathsf{ANN}(n,d,t,\epsilon)$ .
4.3 Hardness Result
In the remainder of this section, we prove our lower bound for attention computation:
Theorem 4.6 (Main Result, formal version of Theorem 1.3). Assuming SETH, for every sufficiently small  $q > 0$ , there are constants  $C > 0$  and  $C_{\alpha} > 0$  and  $C_{\beta} > 1$  such that Approximate Attention Computation AAttC (Definition 1.2) for parameters  $(n,d = C\log n,B = C_{\beta}\sqrt{\log n},\epsilon_a = n^{-C_\alpha})$  requires  $\Omega (n^{2 - q})$  time.
Proof. This follows from combining Lemma 4.3 (hardness for approximation nearest neighbor search) and Lemma 4.7 (a reduction from approximate nearest neighbor search to approximate attention computation) which we prove below.  $\square$
Lemma 4.7. For any constant  $C_{\gamma} \in (0,0.1)$ : For every  $\epsilon > 0$  and  $C > 0$ , there exist constants  $C_a > 0$  and  $C_b > 0$  and such that, if AAttC (Definition 1.2) for parameters  $(2n, d = 2C \log n, B = C_b \sqrt{\log n}, \epsilon_a = n^{-C_a})$  can be solved in time  $T$ , then Gap-ANN  $(n, d = C \log n, t, \epsilon)$  (Definition 4.5) can be solved in time  $O(T + n^{2 - C_{\gamma}})$ .
Due to space limitation, we defer the proof of Lemma 4.7 to Appendix B.
In this work, we showed that how quickly one can perform attention computation depends critically on the magnitude,  $B$ , of the entries of the input matrices. Our main idea was to make use of similarities between attention computation and KDE, and to show how many known techniques for KDE can also be used in this setting. Since KDE is a very well-studied problem, it would be exciting to see what other techniques can be applied to attention computation as well. One limitation of our lower bound result is, it is a conditional lower bound which is based on a well-known conjecture SETH in the area of complexity. It would be interesting to show unconditional lower bound for future work. As far as we are aware, our work does not have negative societal impacts.
Acknowledgements Josh Alman was partly supported by a grant from the Simons Foundation (Grant Number 825870 JA). The authors would like to thank Beidi Chen for helpful discussions related to LLMs, and Feyza Duman, Chinmay Hegde, and Piotr Indyk for helpful comments on an earlier draft. The authors would like to appreciate very constructable feedbacks for NeurIPS 2023 Reviewers. The authors would like to thanks Lichen Zhang and Ruizhe Zhang for useful and helpful suggestions about proof-reading the paper.
[AA22] Amol Aggarwal and Josh Alman. Optimal-degree polynomial approximations for exponentials and gaussian kernel density estimation. In 37th Computational Complexity Conference (CCC 2022). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2022.
[ACM+20] Josh Alman, Timothy Chu, Gary Miller, Shyam Narayanan, Mark Sellke, and Zhao Song. Metric transforms and low rank matrices via representation theory of the real hyperrectangle. arXiv preprint arXiv:2011.11503, 2020.
[ACSS20] Josh Alman, Timothy Chu, Aaron Schild, and Zhao Song. Algorithms and hardness for linear algebra on geometric graphs. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), pages 541-552. IEEE, 2020.
[AS23] Josh Alman and Zhao Song. How to capture higher-order correlations? generalizing matrix softmax attention to kronecker computation. arXiv preprint arXiv:2310.04064, 2023.
[BCIS18] Arturs Backurs, Moses Charikar, Piotr Indyk, and Paris Siminelakis. Efficient density evaluation for smooth kernels. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 615-626. IEEE, 2018.
[BIS17] Arturs Backurs, Piotr Indyk, and Ludwig Schmidt. On the fine-grained complexity of empirical risk minimization: Kernel methods and neural networks. Advances in Neural Information Processing Systems, 30, 2017.
$\left[\mathrm{BMR}^{+}20\right]$  Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877-1901, 2020.
$\left[\mathrm{CDL}^{+}22\right]$  Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations (ICLR), 2022.
$\left[\mathrm{CDW}^{+}21\right]$  Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems (NeurIPS), 34:17413-17426, 2021.
[CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
[CKNS20] Moses Charikar, Michael Kapralov, Nvid Nouri, and Paris Siminelakis. Kernel density estimation through density constrained near neighbor search. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), pages 172-183. IEEE, 2020.
$\left[\mathrm{CND}^{+}22\right]$  Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[CS17] Moses Charikar and Paris Siminelakis. Hashing-based-estimators for kernel density in high dimensions. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pages 1032–1043. IEEE, 2017.
[RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[SM19] Karthik C. S. and Pasin Manurangsi. On closest pair in euclidean metric: Monochromatic is as hard as bichromatic. 10th Innovations in Theoretical Computer Science (ITCS), 2019.
[TBY+19] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4344–4353, 2019.
$\left[\mathrm{VSP}^{+}17\right]$  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 30, 2017.
[Wil18] Virginia Vassilevska Williams. On some fine-grained questions in algorithms and complexity. In Proceedings of the international congress of mathematicians: Rio de janeiro 2018, pages 3447-3487. World Scientific, 2018.
[WLK+20] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
[ZBIW19] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36-39. IEEE, 2019.
[ZHDK23] Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. Kdeformer: Accelerating transformers via kernel density estimation. In ICML. arXiv preprint arXiv:2302.02451, 2023.
$\left[\mathrm{ZRG}^{+}22\right]$  Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[ZSZ+23] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. In NeurIPS. arXiv preprint arXiv:2306.14048, 2023.
Roamdap.
We provide the missing proofs of our upper bound (algorithm) result in Section A. We provide the missing proofs of our lower bound (hardness) result in Section B.
A Missing Proofs for Upper Bound
A.1 Proof of Lemma 3.2
Lemma A.1 (Restatement of Lemma 3.2). Let  $M = XY^{\top} \in \mathbb{R}^{n \times n}$  denote a matrix with  $X, Y \in \mathbb{R}^{n \times d}$ . Let  $P(x)$  denote a degree- $g$  polynomial, and define  $r = \binom{2(g+d)}{2g}$ .
Proof. Let  $P(x)$  denote the degree- $g$  polynomial. Expand it in terms of its coefficients as
$$
P (x) = \sum_ {i = 0} ^ {d} c _ {i} \cdot x ^ {i}.
$$
Consider the function  $\mathsf{K} : \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$  defined by, for  $u, v \in \mathbb{R}^d$ ,
$$
\mathsf {K} (u, v) := P (\langle u, v \rangle).
$$
$\mathsf{K}$  is a degree-  $2g$  polynomial in the  $2d$  entries  $u_{1},\dots u_{d},v_{1},\dots ,v_{d}$  of the vectors  $u,v$ . Define the set  $V$  of its variables,
$$
V := \left\{u _ {1}, \dots , u _ {d}, v _ {1}, \dots , v _ {d} \right\}.
$$
Let  $\mathcal{F}$  denote the set of functions
$$
\mathcal {F} := \left\{f: V \to \{0, 1, 2, \dots , 2 g \} \mid \sum_ {v \in V} f (v) \leq 2 g \right\}.
$$
We can count that
$$
| \mathcal {F} | = \left( \begin{array}{c} 2 d + 2 g \\ 2 g \end{array} \right).
$$
Hence, there are coefficients  $c_{t}\in \mathbb{R}$  for each  $t\in \mathcal{F}$  such that
$$
\mathsf {K} (u, v) = \sum_ {t \in \mathcal {F}} c _ {t} \cdot \prod_ {v \in V} v ^ {t (v)}.
$$
$$
V _ {u} := \{u _ {1}, \dots , u _ {d} \}
$$
$$
V _ {v} = V \backslash V _ {u}.
$$
We define  $\phi_u: \mathbb{R}^d \to \mathbb{R}^{|\mathcal{F}|}$  by, for any  $t \in \mathcal{F}$ ,
$$
\phi_ {u} (u _ {1}, \dots , u _ {d}) _ {t} = c _ {t} \cdot \prod_ {v _ {i} \in V _ {u}} u _ {i} ^ {t (u _ {i})}.
$$
Similarly, we define  $\phi_v: \mathbb{R}^d \to \mathbb{R}^{|\mathcal{F}|}$  by, for any  $t \in \mathcal{F}$ ,
$$
\phi_ {v} (v _ {1}, \dots , v _ {d}) _ {t} = \prod_ {v _ {i} \in V _ {v}} v _ {i} ^ {t (u _ {i})}.
$$
$$
\mathsf {K} (u, v) = \langle \phi_ {u} (u), \phi_ {v} (v) \rangle .
$$
For  $i \in [n]$ , let  $X_{i} \in \mathbb{R}^{d}$  denote the  $i$ -th row of  $X$ , and let  $Y_{i} \in \mathbb{R}^{d}$  denote the  $i$ -th row of  $Y$ . Our algorithm can thus construct
- the matrix  $U_{1}\in \mathbb{R}^{n\times |\mathcal{F}|}$  whose  $i$ -th row is the vector  $\phi_u(x_i)$  for  $i\in [n]$ , and
- the matrix  $U_{2} \in \mathbb{R}^{n \times |\mathcal{F}|}$  whose  $i$ -th row is the vectors  $\phi_v(y_i)$  for  $i \in [n]$ .
Each entry of these matrices can be constructed by multiplying together at most  $g$  variables, so these  $n \times r$  matrices can be constructed in time  $O(nrg)$  as desired.
A.2 Proof of Lemma 3.5
Lemma A.2 (Restatement of Lemma 3.5). Let  $A \in \mathbb{R}^{n \times n}$  be any matrix whose entries are all positive and  $\epsilon_A \in (0,0.1)$  be any parameter. Let  $\widetilde{A} \in \mathbb{R}^{n \times n}$  be any matrix such that, for all  $(i,j) \in [n] \times [n]$ , we have
Proof. We calculate that
$$
\begin{array}{l} | \widetilde {D} _ {i, i} - D _ {i, i} | = \left| \sum_ {j = 1} ^ {n} \widetilde {A} _ {i, j} - \sum_ {j = 1} A _ {i, j} \right| \\ \leq \sum_ {j = 1} ^ {n} | \widetilde {A} _ {i, j} - A _ {i, j} | \\ \leq \sum_ {j = 1} ^ {n} \epsilon_ {A} A _ {i, j} \\ = \epsilon_ {A} \cdot D _ {i}. \\ \end{array}
$$
where the second step follows from triangle inequality.
A.3 Proof of Lemma 3.6
Lemma A.3 (Restatement of Lemma 3.6). Let  $\epsilon_A, \epsilon_D \in (0, 0.1)$  and  $B > 1$  be parameters, and let  $V \in \mathbb{R}^{n \times d}$  denote a matrix with  $\|V\|_{\infty} \leq B$ . Let  $A \in \mathbb{R}^{n \times n}$  be any matrix whose entries are all positive, and let  $\widetilde{A} \in \mathbb{R}^{n \times n}$  be a matrix such that, for all  $(i, j) \in [n] \times [n]$  we have
$$
\| \widetilde {D} ^ {- 1} \widetilde {A} V - D ^ {- 1} A V \| _ {\infty} \leq \| \widetilde {D} ^ {- 1} \widetilde {A} V - D ^ {- 1} \widetilde {A} V \| _ {\infty} + \| D ^ {- 1} \widetilde {A} V - D ^ {- 1} A V \| _ {\infty}. \tag {1}
$$
We now bound each of these two terms separately.
First, for each  $(i,j)\in [n]\times [d]$
$$
\begin{array}{l} \left| \left(\widetilde {D} ^ {- 1} \widetilde {A} V - D ^ {- 1} \widetilde {A} V\right) _ {i, j} \right| = \left| \sum_ {l = 1} ^ {n} \left(\widetilde {D} _ {i, i} ^ {- 1} - D _ {i, i} ^ {- 1}\right) \cdot \widetilde {A} _ {i, l} \cdot V _ {l, j} \right| \\ \leq \sum_ {l = 1} ^ {n} | (\widetilde {D} _ {i, i} ^ {- 1} - D _ {i, i} ^ {- 1}) \cdot \widetilde {A} _ {i, l} | \cdot \| V \| _ {\infty} \\ = \sum_ {l = 1} ^ {n} | \frac {D _ {i , i} - \widetilde {D} _ {i , i}}{D _ {i , i} \widetilde {D} _ {i , i}} \widetilde {A} _ {i, l} | \cdot \| V \| _ {\infty} \\ \leq \epsilon_ {D} \cdot \sum_ {l = 1} ^ {n} | \widetilde {D} _ {i} ^ {- 1} \widetilde {A} _ {i, l} | \cdot \| V \| _ {\infty} \\ = \epsilon_ {D} \cdot | \sum_ {l = 1} ^ {n} \widetilde {D} _ {i} ^ {- 1} \widetilde {A} _ {i, l} | \cdot \| V \| _ {\infty} \\ = \epsilon_ {D} \cdot \| V \| _ {\infty} \\ \leq \epsilon_ {D} \cdot B \tag {2} \\ \end{array}
$$
where the second step follows from the triangle inequality, the forth step follows from  $|(D_{i,i} - \widetilde{D}_{i,i}) / D_{i,i}| \leq \epsilon_D$ , the fifth step follows from  $\widetilde{D}_i^{-1} > 0$  and  $\widetilde{A}_{i,l} > 0$ , and the last step follows from our assumption on  $V$ .
Second, for each  $(i,j)\in [n]\times [d]$
$$
\begin{array}{l} | (D ^ {- 1} \widetilde {A} V - D ^ {- 1} A V) _ {i, j} | = | \sum_ {l = 1} ^ {n} D _ {i, i} ^ {- 1} (\widetilde {A} _ {i, l} - A _ {i, l}) \cdot V _ {l, j} | \\ \leq \sum_ {l = 1} ^ {n} | D _ {i, i} ^ {- 1} | \cdot | (\widetilde {A} _ {i, l} - A _ {i, l}) | \cdot \| V \| _ {\infty} \\ = \sum_ {l = 1} ^ {n} D _ {i, i} ^ {- 1} \cdot | (\widetilde {A} _ {i, l} - A _ {i, l}) | \cdot \| V \| _ {\infty} \\ \leq \sum_ {l = 1} ^ {n} D _ {i, i} ^ {- 1} \cdot \epsilon_ {A} A _ {i, l} \cdot B \\ = \epsilon_ {A} \cdot B, \tag {3} \\ \end{array}
$$
where the second step follows from triangle inequality, the third step follows from  $D_{i,i}^{-1} > 0$ , the forth step follows from  $|\widetilde{A}_{i,l} - A_{i,l}| \leq \epsilon_A \cdot A_{i,l}$  and the last step follows from definition of  $D_{i,i}$ .
The result follows by combining Eq. (1), and two inequalities (Eq. (2) and Eq. (3)).
B Missing Proofs for Lower Bound
Lemma B.1 (Restatement of Lemma 4.7). For any constant  $C_{\gamma} \in (0,0.1)$ : For every  $\epsilon > 0$  and  $C > 0$ , there exist constants  $C_a > 0$  and  $C_b > 0$  and such that, if AAttC (Definition 1.2) for parameters  $(2n,d = 2C\log n, B = C_b\sqrt{\log n}, \epsilon_a = n^{-C_a})$  can be solved in time  $T$ , then Gap-ANN  $(n,d = C\log n,t,\epsilon)$  (Definition 4.5) can be solved in time  $O(T + n^{2 - C_{\gamma}})$ .
Proof. We give an algorithm with the stated running time for  $\mathsf{Gap - ANN}(n,d = C\log n,t,\epsilon)$ . Let  $c > 0$  be a parameter we will choose later (it will be a function of  $C$  and  $C_{\gamma}$ ). Our algorithm will proceed to one of two cases depending on the value of  $t$ . If  $t < c\log n$ , then we will use one algorithm
which runs in time  $O(n^{2 - C_{\gamma}})$ . Otherwise, if  $t \geq c \log n$ , we will use another algorithm which runs in time  $O(T)$ .
Case 1:  $t < c \log n$ .
Let  $a_1, \dots, a_n, b_1, \dots, b_n \in \{0, 1\}^d$  be the input vectors to Gap-ANN, and let  $t \in [0, d]$  denote the target distance. Recall that  $d = C \log n$ .
In this  $t < c \log n$  case, we will simply brute-force for the answer in the following way: We first store the vectors  $b_{1}, \dots, b_{n}$  in a lookup table, then for each  $i \in [n]$ , we iterate over all vectors  $b' \in \{0, 1\}^{d}$  which have Hamming distance at most  $t$  from  $a_{i}$  and check whether  $b'$  is in the lookup table. This determines whether there is a  $b \in B$  at distance at most  $t$  from  $a_{i}$ , as desired.
For each  $i \in [n]$ , we need to iterate over  $\binom{d}{t}$  choices for the vector  $b'$ , so the total running time will be  $O(n \cdot \binom{d}{t})$ . By standard bounds on binomial coefficients, we know that
$$
\begin{array}{l} n \cdot \left( \begin{array}{c} d \\ t \end{array} \right) \leq n \cdot \left( \begin{array}{c} C \log n \\ c \log n \end{array} \right) \\ \leq n ^ {1 + f (C, c)} \\ \end{array}
$$