SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tamas Sarlos
stamas@google.com
Xingyou Song
xingyousong@google.com
Inspired by fast algorithms in natural language processing, we study low rank approximation in the entrywise transformed setting where we want to find a good rank  $k$  approximation to  $f(U \cdot V)$ , where  $U, V^{\top} \in \mathbb{R}^{n \times r}$  are given,  $r = O(\log(n))$ , and  $f(x)$  is a general scalar function. Previous work in sublinear low rank approximation has shown that if both (1)  $U = V^{\top}$  and (2)  $f(x)$  is a PSD kernel function, then there is an  $O(nk^{\omega - 1})$  time constant relative error approximation algorithm, where  $\omega \approx 2.376$  is the exponent of matrix multiplication. We give the first conditional time hardness results for this problem, demonstrating that both conditions (1) and (2) are in fact necessary for getting better than  $n^{2 - o(1)}$  time for a relative error low rank approximation for a wide class of functions. We give novel reductions from the Strong Exponential Time Hypothesis (SETH) that rely on lower bounding the leverage scores of flat sparse vectors and hold even when the rank of the transformed matrix  $f(UV)$  and the target rank are  $n^{o(1)}$ , and when  $U = V^{\top}$ . Furthermore, even when  $f(x) = x^{p}$  is a simple polynomial, we give runtime lower bounds in the case when  $U \neq V^{\top}$  of the form  $\Omega(\min(n^{2 - o(1)}, \Omega(2^{p})))$ . Lastly, we demonstrate that our lower bounds are tight by giving an  $O(n \cdot \mathrm{poly}(k, 2^{p}, 1 / \epsilon))$  time relative error approximation algorithm and a fast  $O(n \cdot \mathrm{poly}(k, p, 1 / \epsilon))$  additive error approximation using fast tensor-based sketching. Additionally, since our low rank algorithms rely on matrix-vector product subroutines, our lower bounds extend to show that computing  $f(UV)W$ , for even a small matrix  $W$ , requires  $\Omega(n^{2 - o(1)})$  time.
The central idea behind the classic problem of low rank approximation (LRA) is to approximate a given matrix with a rank  $k$  matrix that preserves the important features of the original matrix, while being computationally more efficient and statistically more stable. One particular area of interest in LRA is the low rank decomposition of entrywise transformed matrices, where the entries of the original matrix are transformed through a non-linear function before being approximated, with various applications in kernel methods, self-attention, and likelihood computations [Choromanski et al., 2021, Levy and Goldberg, 2014]. In these settings, the Gram matrix of a dot-product kernel, the attention module's product operator, or even the non-linear computation of activation in a deep network can all be represented as  $f(UV)$ , where the inputs are low dimensional matrices  $U, V$ . Note that while  $UV$  is low rank,  $f(UV)$  may not be, with the rank blowup dependent on the choice of the transformation  $f$ .
For entrywise transformed low rank approximation, we aim to find an approximately optimal rank  $k$  approximation, in terms of relative Frobenius norm error, to the matrix  $A = f(UV)$  given  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$ , where  $f$  is a scalar transformation. In this paper, our goal is to study the functions  $f$  such that this task is solvable in subquadratic  $O(n^{2 - \epsilon})$  time and for ease of presentation, we assume  $n = d$  in this section. This problem was studied in the distributed setting in Woodruff and Zhong [2016] for functions  $f(x) = |x|^p$ , where the goal was to minimize communication. It was also studied in the streaming setting, where it was shown that  $O(n \text{ poly}(k / \epsilon))$  memory suffices to solve rank  $k$  approximation with additive error  $O(\epsilon \| U \| \| V \|)$  for the function  $f(x) = \log(|x| + 1)$  in a single pass [Jiang et al., 2021], improving the earlier work of [Liang et al., 2020]. In a related paper, Han et al. [2020] present an algorithm for low-rank approximation of polynomial entrywise transforms; however, they make no comparison to the optimal rank  $k$  error.
In the setting when  $U = V^{\top}$ , certain choices of  $f(x)$  can surprisingly admit subquadratic relative error algorithms, such as when  $f$  represents a positive semi-definite (PSD) kernel [Musco and Musco, 2017]. Moreover, for any PSD matrix of any rank, by sampling according to the leverage scores of the matrix square root of the kernel, recent work shows that relative error low rank decomposition is possible in  $O(n(k / \epsilon)^{\omega -1})$  time, where  $\omega \approx 2.373$  is the exponent of matrix multiplication; see [Musco and Woodruff, 2017, Bakshi et al., 2020]. Note that in many cases when  $U = V^{\top}$ , carefully choosing  $f$  will result in  $A = f(UV)$  being PSD. In fact, when  $u_{i}$  are all unit norm, if  $f$  admits a Taylor expansion with non-negative coefficients, then  $A = f(UU^{\top})$  is a PSD matrix and admit sub-quadratic low rank approximations.
On the surface, this astounding algorithmic result seems to conflict with subquadratic lower bounds for basic linear algebraic primitives for many kernel functions, especially the exponential kernel used in the attention architecture. Specifically, such work looks at kernel matrix vector products and shows they take  $\Omega(n^{2 - o(1)})$  time for relative and additive error approximations [Keles et al., 2023]. Furthermore, these bounds can be refined to hold in the restricted regime when the entries of the matrix are  $\Omega(\sqrt{\log(n)})$  [Alman and Song, 2023]. These conditional lower bounds, as with others for linear algebraic problems, are derived from hardness based on the Strong Exponential Time Hypothesis (SETH) [Lokshtanov et al., 2013].
The crucial observation to resolve this seeming contradiction is to recognize that the low rank approximation allows for an error bound that depends on the error of the best rank- $k$  approximation, which can be much larger than the error tolerated in the previous lower bounds. Therefore, hardness for LRA is inherently different than hardness for approximating the entire matrix additively or than matrix-vector (MV) multiplication. In fact, we will show that hardness for MV multiplication is, in some sense, strictly easier to establish.
Some related lower bounds include the work of Backurs et al. [2017] that solving kernel Support Vector Machines (SVM), ridge regression, or Principal Component Analysis (PCA) problems to high accuracy or approximating kernel density estimates up to a constant factor for kernels with exponential tails, requires  $n^{2 - o(1)}$  time assuming SETH. Also, Alman et al. [2020] show that for linear algebraic primitives for the Laplacian of a graph with weights given by a kernel, most operations, such as  $\epsilon$ -approximate matrix-vector multiplication, are hard, although their hardness results assume a  $\log(1/\epsilon)$  dependence when reducing to SETH.
In the setting of entrywise-transformed low rank approximation, we show that we cannot get subquadratic relative error LRA generally when either 1)  $U \neq V^{\top}$  even for PSD kernel functions or 2) for transformations that are approximately polynomials of  $|x|$  even for constant degree. Therefore, without multiple strong structural assumptions on  $A$ , there is no subquadratic approximation algorithm, even when  $r = \Theta(\log(n))$  and the rank of the transformed matrix  $f(UV)$  and the target rank are  $n^{o(1)}$ . We emphasize these novel hardness results hold for a large class of transformations and in fact generalize to hardness for matrix vector multiplication, which can be derived as a corollary, implying that computing  $f(UV)z$  also requires  $\Omega(n^{2 - o(1)})$  time for many  $f$  that have not been studied before (see Theorem 3.2). On the positive side, for the polynomial activation  $f(x) = x^p$ , we provide an  $O(n \cdot \mathrm{poly}(r^p, k, 1/\epsilon))$  time algorithm for relative error LRA and  $O(n \cdot \mathrm{poly}(p, k, 1/\epsilon))$  time for additive error LRA via fast tensor-based sketches (see Algorithm 2).
Upper Bounds
Prior Work	Complexity	Matrix Type	Task
Musco and Woodruff [2017]	O(n·poly(k/ε))	PSD, U = V†	Relative LRA
Bakshi et al. [2020]	O(n·(k/ε)ω-1)	PSD, U = V†	Relative LRA
This work	O(n·poly(2p, k, 1/ε))	f(x) = xp, U ≠ V†	Relative LRA
This work	O(n·poly(p, k, 1/ε))	f(x) = xp, U ≠ V†	Additive LRA
Lower Bounds
Backurs et al. [2017]	Ω(n2-o(1))	Gaussian, U = V†	PCA, Regression
Alman et al. [2020]	Ω(n2-o(1))	Kernel Laplacians U = V†	MV product
Keles et al. [2023]	Ω(n2-o(1))	Exponential Kernels, U = V†	MV product
This work	Ω(n2-o(1))	f(x) = |x|p + O(|x|p+1), U = V††	Relative LRA, MV
This work	Ω(min(n2-o(1), 2Ω(p)))	f(x) = xp, U ≠ V†	Relative LRA, MV
Table 1: Overview of upper bounds for low rank approximation (LRA) and lower bounds for relative linear algebraic primitives, where the dependence on  $r$  is omitted. This table illustrates multiple separation results: 1) LRA is strictly easier than matrix vector (MV) products for positive semidefinite kernels, 2) LRA is strictly easier for functions  $f(x)$  that are kernels, even when considering low-degree polynomials of  $|x|$ , 3) LRA is strictly easier when  $U = V^{\top}$  for  $f(x) = x^{p}$  for  $p = \Omega(\log(n))$ , which has a  $2^{\Omega(p)}$  lower bound when  $U \neq V^{\top}$ , 4) additive error LRA is strictly easier than relative error LRA. Our work extends to any function  $f(x)$  that admits a Taylor series dominated by  $|x|^{p}$  around  $x = 0$ . For example, this includes  $f(x) = \log(|x| + 1) = |x| + O(|x|^2)$ .
We note that previous lower bound techniques do not apply for LRA since relative error LRA approximations of  $f(UV)$  are possible without approximating a matrix vector product, as shown by the  $O(n^{1 + o(1)})$  time LRA algorithm for the popular exponential kernel  $f(x) = \exp (x)$  by querying a sublinear number of entries of  $f(UU^{\top})$  [Musco and Woodruff, 2017, Bakshi et al., 2020]. Indeed, our novel lower bounds differ from previous reductions from Orthogonal Vectors Problem (OVP) by explicitly creating LRA instances and exploiting linear algebraic structural properties of the column spaces of a low-rank tensored matrix.
Specifically, our reduction uses an structural property that if there is a pair of input vectors to OVP which are orthogonal, then under mild assumptions, this implies we can find a sparse vector in the column span of the low rank approximation of  $f(UV)$ , where  $U, V$  are matrices containing the input vectors to OVP. Also, our algorithm (Algorithm 1) relies on another critical observation that this sparse vector has uniform magnitude in the non-zero entries, each of which corresponds to a vector with an orthogonal vector pair. By exploiting the structure of this vector, we can lower bound the leverage score of each row that corresponds to a non-zero entry of this sparse vector, and we can appeal to fast leverage score computation algorithms to find a small  $n^{o(1)}$ -size superset of the support of the sparse vector. Finally, we can quickly check which pairs of entries in the superset correspond to vectors in the original OVP problem via a brute-force search, completing the reduction. The precise reduction is a bit more technically involved, as we also need to allow for additive error for our applications.
In the setting when  $U \neq V^{\top}$ , we show novel lower bounds in the case when  $f(x) = x^{p}$  is a polynomial kernel function and admits a simple rank  $r^p$  decomposition. We show that we cannot do better than the naive decomposition and prove an  $\Omega(\min(n^{2 - o(1)}, 2^{\Omega(p)})$  lower bound (see Theorem 3.3). This implies in the setting when  $p = \omega(\log(n))$  and  $k = n^{o(1)}$ , that surprisingly there is a separation between the (1)  $U = V^{\top}$  setting and the (2)  $U \neq V^{\top}$  setting: when  $f(x) = x^{2p}$ , in the first setting, previous works show that there admits an  $O(nk^{\omega - 1})$  approximation algorithm; however, in the second setting when  $U \neq V^{\top}$ , our lower bounds imply that there cannot exist a truly subquadratic time algorithm. We emphasize that these exponential lower bounds in the degree for the polynomial kernel are the first of their kind to rule out truly subquadratic algorithms for  $p = \omega(\log(n))$ , even when  $f(x) = x^{p}$  is a simple polynomial.
We also give an  $O(n \cdot \mathrm{poly}(r^p / \epsilon))$  time algorithm for relative error approximation, which agrees with our lower bounds (see Theorem 4.1), showing that our lower bounds for the polynomial kernel when  $U \neq V^\top$  are in fact tight. In addition, we provide an  $O(n \cdot \mathrm{poly}(p, k, 1 / \epsilon))$  time algorithm for additive error LRA that avoids the inherent exponential dependence on  $p$ , highlighting a separation between additive and multiplicative error LRA when  $p = \Omega(\log(n))$ . Specifically, we can achieve the easier additive error approximation with fast tensor-based sketching matrices (see Theorem 4.2) by applying comparable techniques to an independent subsequent work on polynomial-based transformers, although the main difference is that they do not output a rank  $k$  approximation and suffers a worse dependence on  $p, 1 / \epsilon$  due to their non-negativity guarantees [Kacham et al.,
2023]. We remark that while the polynomial tensor matrix itself can be approximated relatively in  $O(n \cdot \mathrm{poly}(p))$  runtime, we emphasize that the polynomial kernel, as a product of two tensor matrices, must incur additive error or suffer  $\Omega(2^p)$  runtime for relative error guarantees.
Finally, we observe that we can generalize our lower bounds for LRA by exploiting the fact that our algorithms for fast LRA reduce to matrix vector product! Therefore, our relative error algorithms are reductions that give MV lower bounds, which are automatically derived for a large number of entrywise transformed matrices, even when  $U = V^{\top}$  (see Theorem 4.3). This implies that in some sense, LRA is an easier problem than MV approximation, implying that our lower bounds for LRA are stronger. We summarize the most relevant prior results and our contributions in Table 1.
1.2 Related Work to Transformers and Natural Language Processing
A primary downstream application from our work is in the field of natural language, as it is common to compute similarity matrices  $f(UV)$  from token embeddings  $U, V$ . Such is the case for the popular use of Transformers [Vaswani et al., 2017] and their efficient attention variants [Tay et al., 2022]. The standard attention mechanism consists of the operation  $\text{softmax}(\frac{UV}{\sqrt{d}})W$ , where in our notation,  $U, V, W$  are the "query", "key", and "value" matrices respectively. Simplifying and ignoring normalizations, this can be seen as  $f(UV)W$  where  $f(x) = \exp(x)$ . For a given  $f$ , one can potentially linearize the attention mechanism if there exist  $U^{\star}, V^{\star}$  such that  $f(UV) \approx U^{\star}V^{\star}$ , as then the order of matrix multiplication can be rearranged into  $U^{\star}(V^{\star}W)$  which allows memory and runtime in  $O(ndk)$ .
In the unnormalized softmax case where  $f(x^{\top}y) = \exp (x^{\top}y)$ , Choromanski et al. [2021] notes that  $\exp (x^{\top}y) = \mathbb{E}_{\zeta \sim \mathcal{N}(0,\mathbf{I}_d)}\left[\exp \left(\zeta^{\top}x - \frac{\|x\|^2}{2}\right)\exp \left(\zeta^{\top}y - \frac{\|y\|^2}{2}\right)\right]$  which thus allows defining  $U^{\star}, V^{\star}$  as random feature matrices from sampled  $\zeta_1, \ldots, \zeta_k$ . Further use of kernel properties to improve the softmax approximation have been introduced in Choromanski et al. [2022], Likhosherstov et al. [2022]. More generally, for  $f$  such that  $f(x^{\top}y) = \mathcal{K}(x,y)$  admits a kernel representation, one may consider using variants of Bochner's theorem [Feller, 1968] to provide approximations via random Fourier features [Rahimi and Recht, 2007].
While works such as Tsai et al. [2019], Kacham et al. [2023] have experimented with linear, polynomial, exponential, and RBF kernels, so far the dominant paradigm, termed the class of Linear Transformers Katharopoulos et al. [2020], is to conveniently instead consider the reverse case, where one defines a mapping  $\phi : \mathbb{R} \to \mathbb{R}$  in order to define the kernel  $\mathcal{K}(x,y) = \phi(x)^{\top} \phi(y)$ . Unfortunately, usually this does not lead to a closed-form  $f$  such that  $f(x^{\top}y) = \phi(x)^{\top} \phi(y)$ , which can lack interpretability and compatibility with classic attention mechanisms.
However, one may consider nonlinear functions  $f$  which do not admit a kernel; for example, the class of functions  $f(x) = \log^c(|x| + 1)$  for  $c > 0$  is used in Levy and Goldberg [2014], Li et al. [2015] to compute implicit word embeddings and corresponding generative models. Within this application domain, our work thus provides answers to the question: for which classes of functions  $f$  can one efficiently compute low-rank approximations  $U^\star$ ,  $V^\star$  such that  $f(UV) \approx U^\star V^\star$ , especially when  $f(x^\top y)$  does not admit a kernel structure  $\mathcal{K}(x, y)$ ?
We let our implicit matrix  $A = f(UV)$  be  $n \times d$  where  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$  and  $n \geq d$  without loss of generality, where  $f: \mathbb{R} \to \mathbb{R}$  is a scalar function. We will also assume that  $d = n^{\Omega(1)}$  and it is often the case in empirical settings that  $d = n$ . We say that a matrix  $A$  is positive semi-definite (PSD) if it is symmetric and only has non-negative eigenvalues; a function  $f$  is a kernel function if  $f(UU^\top)$  is PSD for any matrix  $U$ . The  $i$ -th leverage score  $\ell_i$  of matrix  $B \in \mathbb{R}^{n \times d}$  is equal to its sensitivity, i.e.  $\ell_i = \sup_{x \in \mathbb{R}^d} \frac{(B_i^T x)^2}{\|Bx\|_2^2} = \sup_{y \in \operatorname{colspan}(B)} \frac{y_i^2}{\|y\|_2^2}$ , where  $B_i$  is the  $i$ -th row of  $B$ . The Khatri-Rao product of  $B \in \mathbb{R}^{n \times d}$  is  $C \in \mathbb{R}^{n \times d^p}$ , where the  $i$ -th row of  $C$  is  $B_i$  tensored with itself  $p$  times. We use the standard notation that  $\tilde{O}(f)$  is  $O(f \text{ poly}(\log f))$ .
For the approximate rank  $k$  LRA problem, we want to find  $U^{\prime}\in \mathbb{R}^{n\times k}$  and  $V^{\prime}\in \mathbb{R}^{k\times n}$  such that
$$
\| A - U ^ {\prime} V ^ {\prime} \| \leq (1 + \epsilon) \min  _ {\tilde {U} \in \mathbb {R} ^ {n \times k}, \tilde {V} \in \mathbb {R} ^ {k \times d}} \| A - \tilde {U} \tilde {V} \|
$$
where  $\| \cdot \|$  denotes the Frobenius norm, unless otherwise specified and we denote  $[A]_k = \tilde{U}^\star \tilde{V}^\star$  as some rank  $k$  matrix that minimizes the objective. Note this approximation is a relative error approximation guarantee, but can be analogously defined for additive error. For stronger lower bounds, we consider a weakened version of this problem given by outputting only a best rank  $k$  projection, specifically we want to find orthogonal  $W \in \mathbb{R}^{n \times k}$ :  $\| A - AWW^\top \| \leq (1 + \epsilon) \| A - [A]_k \|$ .
Our lower bounds will rely on reductions from the conditional hardness of OVP and Max-IP, whose hardness comes from SETH. We observe, from our remarks, that some more restrictive structure can be placed on the input vector sets  $A$ ,  $B$  to OVP without removing the hard instances.
Assumption 1. (Hardness of Orthogonal Vectors Problem (OVP)[Williams, 2005]). Let  $A = \{a_1, \ldots, a_n\}$  and  $B = \{b_1, \ldots, b_d\}$  be sets, where  $a_i, b_j \in \{0, 1\}^r$  are binary vectors for all  $i \in [n] = \{1, 2, \ldots, n\}$ ,  $j \in [d]$ . Any algorithm which given any input  $(A, B)$  decides with constant probability if there is at least one pair of vectors  $a \in A$  and  $b \in B$  such that  $a^T b = 0$  requires  $(nd)^{1 - o(1)}$  time, provided  $r = \omega(\log n)$  and  $r = n^{o(1)}$ . Observe that the lower bounds also hold when  $A = B$  is enforced.
Remark 2. Previous work [Williams, 2005, Vassilevska Williams, 2015] has shown that Assumption 1 is true for  $n = d$  unless the Strong Exponential Time Hypothesis (SETH, Impagliazzo and Paturi [2001]) is false. Given this assumption, one can handle general  $n$  and  $d$  by a padding argument: if one could solve OVP with an algorithm  $\mathcal{A}$  running in at most  $(nd)^{1 - C}$  time for a constant  $C > 0$ , and without loss of generality  $n \geq d$ , then one could solve the problem when  $|A| = |B| = n$  by splitting  $B$  into  $\Theta(n / d)$  disjoint sets each of size at most  $d$ , and solving the problem on each disjoint set in total time less than  $(nd)^{1 - C} \cdot \Theta(n / d) = n^{2 - C} / d^C \leq n^{2 - C}$ , contradicting the assumption in the  $|A| = |B| = n$  case.
Remark 3. In Assumption 1 we can enforce more restrictive structure on the input sets  $A, B$  without making the problem easier. Specifically, we can assume that there are at most  $n^{o(1)}$  distinct pairs with  $a \in A$  and  $b \in B$  for which  $a^T b = 0$ . Indeed, otherwise by sampling  $(nd)^{2 - \Omega(1)}$  pairs at random and checking if  $a^T b = 0$ , we would solve the OVP problem in  $(nd)^{1 - \Omega(1)} \cdot r = (nd)^{1 - \Omega(1)}$  time, using that  $r = n^{o(1)}$ . This would contradict Assumption 1.
Assumption 4. (Hardness of  $\text{Apx-Max-IP}_{n,d}$  Problem, Definition 2.1, Remark 2.2, and Lemma 4.1 of Chen and Williams [2019]). Two sets  $A = \{a_1, \ldots, a_n\}$  and  $B = \{b_1, \ldots, b_n\}$  are given, where  $a_i, b_i \in \{0,1\}^s$  are binary vectors for all  $i \in [n]$ , with  $s = \omega(\log(n))$ . Let  $m = \max_{a \in A, b \in B} a \cdot b$ . Any algorithm which outputs a number  $\tilde{m} \in [m/100, m]$  with constant probability requires  $(nd)^{1 - o(1)}$  time.
Remark 5. As in Remark 2 and Remark 3, we can reduce the general  $n$  and  $d$  case to the case  $n = d$  of previous work [Chen and Williams, 2019], and we can also enforce that there are at most  $n^{o(1)}$  pairs  $a \in A$  and  $b \in B$  for which  $a \cdot b \geq m / 100$ , as otherwise sampling would solve the problem in less than  $(nd)^{1 - o(1)}$  time.
3 LRA Runtime Lower Bounds
We show that the complexity of low rank approximation of an entrywise transformed matrix turns out to heavily depend on the type of entrywise transformation. Based on standard complexity assumptions, we show an  $(nd)^{1 - o(1)} = n^{1 + \Omega (1)}$  time lower bound for entrywise function  $f(x) = |x|^p$  for odd integers  $p$  in Theorem 3.1 below, while we show a weaker  $2^{\Omega (p)}$  lower bound for any integer in Theorem 3.3, which becomes  $(nd)^{1 - o(1)}$  for  $p = \Theta (\log n)$ . This is no accident, as we show a matching  $2^{O(p)}n^{1 + o(1)}$  upper bound in Theorem 4.1. Furthermore, our lower bounds extend to functions that are approximately odd-degree polynomials of  $|x|$ , even if the degree is small. Specifically, this includes the commonly used  $f(x) = \log (|x| + 1)\approx |x|$ .
Our proofs use variations of the Orthogonal Vectors Problem (OVP) to create two types of instances of implicit low rank approximation to create a fast  $O(n^{1 + o(1)})$  time algorithm to solve OVP (see Algorithm 1) when given access to a low rank approximation algorithm with constant relative error and small constant additive error guarantees. Let  $A, B$  be the set of input vectors of OVP. Then, in our reduction, if there is a pair of input vectors which are orthogonal, then applying LRA, we can find either 1) a column span deviation of the low rank approximation or 2) a sparse vector in the column span of the low rank approximation. Note that we do not look at the exact column spans, but allow an additive error depending on the additive error of the output.