Title: Hardness of Low Rank Approximation of Entrywise Transformed Matrix Products
Venue: NeurIPS 2023
Accepted: yes
Keywords: Low rank approximation

kernel methods

fine-grained complexity
Abstract: Inspired by fast algorithms in natural language processing, we study low rank approximation in the entrywise transformed setting where we want to find a good rank $k$ approximation to $f(U \cdot V)$, where $U, V^\top \in \mathbb{R}^{n \times r}$ are given, $r = O(\log(n))$, and $f(x)$ is a general scalar function. Previous work in sublinear low rank approximation has shown that if both (1) $U = V^\top$ and (2) $f(x)$ is a PSD kernel function, then there is an $O(nk^{\omega-1})$ time constant relative error approximation algorithm, where $\omega \approx 2.376$ is the exponent of matrix multiplication. We give the first conditional time hardness results for this problem, demonstrating that both conditions (1) and (2) are in fact necessary for getting better than $n^{2-o(1)}$ time for a relative error low rank approximation for a wide class of functions. We give novel reductions from the Strong Exponential Time Hypothesis (SETH) that rely on lower bounding the leverage scores of flat sparse vectors and hold even when the rank of the transformed matrix $f(UV)$ and the target rank are $n^{o(1)}$, and when $U = V^\top$. Furthermore, even when $f(x) = x^p$ is a simple polynomial, we give runtime lower bounds in the case when $U \neq V^\top$ of the form $\Omega(\min(n^{2-o(1)}, \Omega(2^p)))$. Lastly, we demonstrate that our lower bounds are tight by giving an $O(n \cdot \text{poly}(k, 2^p, 1/\epsilon))$ time relative error approximation algorithm and a fast $O(n \cdot \text{poly}(k, p, 1/\epsilon))$ additive error approximation using fast tensor-based sketching. Additionally, since our low rank algorithms rely on matrix-vector product subroutines, our lower bounds extend to show that computing $f(UV)W$, for even a small matrix $W$, requires $\Omega(n^{2-o(1)})$ time.

Hardness of Low Rank Approximation of Entrywise Transformed Matrix Products

Abstract

Inspired by fast algorithms in natural language processing, we study low rank approximation in the entrywise transformed setting where we want to find a good rank k approximation to f ( U · V ) , where U, V ⊤ ∈ R n × r are given, r = O (log( n )) , and f ( x ) is a general scalar function. Previous work in sublinear low rank approximation has shown that if both (1) U = V ⊤ and (2) f ( x ) is a PSD kernel function, then there is an O ( nk ω -1 ) time constant relative error approximation algorithm, where ω ≈ 2 . 376 is the exponent of matrix multiplication. We give the first conditional time hardness results for this problem, demonstrating that both conditions (1) and (2) are in fact necessary for getting better than n 2 -o (1) time for a relative error low rank approximation for a wide class of functions. We give novel reductions from the Strong Exponential Time Hypothesis (SETH) that rely on lower bounding the leverage scores of flat sparse vectors and hold even when the rank of the transformed matrix f ( UV ) and the target rank are n o (1) , and when U = V ⊤ . Furthermore, even when f ( x ) = x p is a simple polynomial, we give runtime lower bounds in the case when U = V ⊤ of the form Ω(min( n 2 -o (1) , Ω(2 p ))) . Lastly, we demonstrate that our lower bounds are tight by giving an O ( n · poly ( k, 2 p , 1 /ϵ )) time relative error approximation algorithm and a fast O ( n · poly ( k, p, 1 /ϵ )) additive error approximation using fast tensor-based sketching. Additionally, since our low rank algorithms rely on matrix-vector product subroutines, our lower bounds extend to show that computing f ( UV ) W , for even a small matrix W , requires Ω( n 2 -o (1) ) time.

1 Introduction

The central idea behind the classic problem of low rank approximation (LRA) is to approximate a given matrix with a rank k matrix that preserves the important features of the original matrix, while being computationally more efficient and statistically more stable. One particular area of interest in LRA is the low rank decomposition of entrywise transformed matrices, where the entries of the original matrix are transformed through a non-linear function before being approximated, with various applications in kernel methods, self-attention, and likelihood computations [Choromanski et al., 2021, Levy and Goldberg, 2014]. In these settings, the Gram matrix of a dot-product kernel, the attention module's product operator, or even the non-linear computation of activation in a deep network can all be represented as f ( UV ) , where the inputs are low dimensional matrices U, V . Note that while UV is low rank, f ( UV ) may not be, with the rank blowup dependent on the choice of the transformation f .

̸

For entrywise transformed low rank approximation, we aim to find an approximately optimal rank k approximation, in terms of relative Frobenius norm error, to the matrix A = f ( UV ) given U ∈ R n × r and V ∈ R r × d , where f is a scalar transformation. In this paper, our goal is to study the functions f such that this task is solvable in subquadratic O ( n 2 -ϵ ) time and for ease of presentation, we assume n = d in this section. This problem was studied in the distributed setting in Woodruff and Zhong [2016] for functions f ( x ) = | x | p , where the goal was to minimize communication. It was also studied in the streaming setting, where it was shown that O ( n poly ( k/ϵ )) memory suffices to solve rank k approximation with additive error O ( ϵ ∥ U ∥∥ V ∥ ) for the function f ( x ) = log( | x | +1) in a single pass [Jiang et al., 2021], improving the earlier work of [Liang et al., 2020]. In a related paper, Han et al. [2020] present an algorithm for low-rank approximation of polynomial entrywise transforms; however, they make no comparison to the optimal rank k error.

In the setting when U = V ⊤ , certain choices of f ( x ) can surprisingly admit subqradratic relative error algorithms, such as when f represents a positive semi-definite (PSD) kernel [Musco and Musco, 2017]. Moreover, for any PSD matrix of any rank, by sampling according to the leverage scores of the matrix square root of the kernel, recent work shows that relative error low rank decomposition is possible in O ( n ( k/ϵ ) ω -1 ) time, where ω ≈ 2 . 373 is the exponent of matrix multiplication; see [Musco and Woodruff, 2017, Bakshi et al., 2020]. Note that in many cases when U = V ⊤ , carefully choosing f will result in A = f ( UV ) being PSD. In fact, when u i are all unit norm, if f admits a Taylor expansion with non-negative coefficients, then A = f ( UU ⊤ ) is a PSD matrix and admit sub-quadratic low rank approximations.

On the surface, this astounding algorithmic result seems to conflict with subquadratic lower bounds for basic linear algebraic primitives for many kernel functions, especially the exponential kernel used in the attention architecture. Specifically, such work looks at kernel matrix vector products and shows they take Ω( n 2 -o (1) ) time for relative and additive error approximations [Keles et al., 2023]. Furthermore, these bounds can be refined to hold in the restricted regime when the entries of the matrix are Ω( √ log( n )) [Alman and Song, 2023]. These conditional lower bounds, as with others for linear algebraic problems, are derived from hardness based on the Strong Exponential Time Hypothesis (SETH) [Lokshtanov et al., 2013].

The crucial observation to resolve this seeming contradiction is to recognize that the low rank approximation allows for an error bound that depends on the error of the best rankk approximation, which can be much larger that the error tolerated in the previous lower bounds. Therefore, hardness for LRA is inherently different than hardness for approximating the entire matrix additively or than matrix-vector (MV) multiplication. In fact, we will show that hardness for MV multiplication is, in some sense, strictly easier to establish.

Some related lower bounds include the work of Backurs et al. [2017] that solving kernel Support Vector Machines (SVM), ridge regression, or Principal Component Analysis (PCA) problems to high accuracy or approximating kernel density estimates up to a constant factor for kernels with exponential tails, requires n 2 -o (1) time assuming SETH. Also, Alman et al. [2020] show that for linear algebraic primitives for the Laplacian of a graph with weights given by a kernel, most operations, such as ϵ -approximate matrix-vector multiplication, are hard, although their hardness results assume a log(1 /ϵ ) dependence when reducing to SETH.

1.1 Our Contributions

In the setting of entrywise-transformed low rank approximation, we show that we cannot get subquadratic relative error LRA generally when either 1) U = V ⊤ even for PSD kernel functions or 2) for transformations that are approximately polynomials of | x | even for constant degree. Therefore, without multiple strong structural assumptions on A , there is no subquadratic approximation algorithm, even when r = Θ(log( n )) and the rank of the transformed matrix f ( UV ) and the target rank are n o (1) . We emphasize these novel hardness results hold for a large class of transformations and in fact generalize to hardness for matrix vector multiplication, which can be derived as a corollary, implying that computing f ( UV ) z also requires Ω( n 2 -o (1) ) time for many f that have not been studied before (see Theorem 3.2). On the positive side, for the polynomial activation f ( x ) = x p , we provide an O ( n · poly ( r p , k, 1 /ϵ )) time algorithm for relative error LRA and O ( n · poly ( p, k, 1 /ϵ )) time for additive error LRA via fast tensor-based sketches (see Algorithm 2).

̸

̸

̸

̸

We note that previous lower bound techniques do not apply for LRA since relative error LRA approximations of f ( UV ) are possible without approximating a matrix vector product, as shown by the O ( n 1+ o (1) ) time LRA algorithm for the popular exponential kernel f ( x ) = exp( x ) by querying a sublinear number of entries of f ( UU ⊤ ) [Musco and Woodruff, 2017, Bakshi et al., 2020]. Indeed, our novel lower bounds differ from previous reductions from Orthogonal Vectors Problem (OVP) by explicitly creating LRA instances and exploiting linear algebraic structural properties of the column spaces of a low-rank tensored matrix.

Specifically, our reduction uses an structural property that if there is a pair of input vectors to OVP which are orthogonal, then under mild assumptions, this implies we can find a sparse vector in the column span of the low rank approximation of f ( UV ) , where U, V are matrices containing the input vectors to OVP. Also, our algorithm (Algorithm 1) relies on another critical observation that this sparse vector has uniform magnitude in the non-zero entries, each of which corresponds to a vector with an orthogonal vector pair. By exploiting the structure of this vector, we can lower bound the leverage score of each row that corresponds to a non-zero entry of this sparse vector, and we can appeal to fast leverage score computation algorithms to find a small n o (1) -size superset of the support of the sparse vector. Finally, we can quickly check which pairs of entries in the superset correspond to vectors in the original OVP problem via a brute-force search, completing the reduction. The precise reduction is a bit more technically involved, as we also need to allow for additive error for our applications.

̸

In the setting when U = V ⊤ , we show novel lower bounds in the case when f ( x ) = x p is a polynomial kernel function and admits a simple rank r p decomposition. We show that we cannot do better than the na¨ ıve decomposition and prove an Ω(min( n 2 -o (1) , 2 Ω( p ) ) lower bound (see Theorem 3.3). This implies in the setting when p = ω (log( n )) and k = n o (1) , that surprisingly there is a separation between the (1) U = V ⊤ setting and the (2) U = V ⊤ setting: when f ( x ) = x 2 p , in the first setting, previous works show that there admits an O ( nk ω -1 ) approximation algorithm; however, in the second setting when U = V ⊤ , our lower bounds imply that there cannot exists a truly subquadratic time algorithm. We emphasize that these exponential lower bounds in the degree for the polynomial kernel are the first of their kind to rule out truly subquadratic algorithms for p = ω (log( n )) , even when f ( x ) = x p is a simple polynomial.

̸

We also give an O ( n · poly ( r p /ϵ )) time algorithm for relative error approximation, which agrees with our lower bounds (see Theorem 4.1), showing that our lower bounds for the polynomial kernel when U = V ⊤ are in fact tight. In addition, we provide an O ( n · poly ( p, k, 1 /ϵ )) time algorithm for additive error LRA that avoids the inherent exponential dependence on p , highlighting a separation between additive and multiplicative error LRA when p = Ω(log( n )) . Specifically, we can achieve the easier additive error approximation with fast tensor-based sketching matrices (see Theorem 4.2) by applying comparable techniques to an independent subsequent work on polynomialbased transformers, although the main difference is that they do not output a rank k approximation and suffers a worse dependence on p, 1 /ϵ due to their non-negativity guarantees [Kacham et al.,

̸

̸

̸

2023]. We remark that while the polynomial tensor matrix itself can be approximated relatively in O ( n · poly ( p )) runtime, we emphasize that the polynomial kernel, as a product of two tensor matrices, must incur additive error or suffer Ω(2 p ) runtime for relative error guarantees.

Finally, we observe that we can generalize our lower bounds for LRA by exploiting the fact that our algorithms for fast LRA reduce to matrix vector product! Therefore, our relative error algorithms are reductions that give MV lower bounds, which are automatically derived for a large number of entrywise transformed matrices, even when U = V ⊤ (see Theorem 4.3). This implies that in some sense, LRA is an easier problem than MV approximation, implying that our lower bounds for LRA are stronger. We summarize the most relevant prior results and our contributions in Table 1.

1.2 Related Work to Transformers and Natural Language Processing

A primary downstream application from our work is in the field of natural language, as it is common to compute similarity matrices f ( UV ) from token embeddings U, V . Such is the case for the popular use of Transformers [Vaswani et al., 2017] and their efficient attention variants [Tay et al., 2022]. The standard attention mechanism consists of the operation softmax ( UV √ d ) W , where in our notation, U, V, W are the 'query', 'key', and 'value' matrices respectively. Simplifying and ignoring normalizations, this can be seen as f ( UV ) W where f ( x ) = exp( x ) . For a given f , one can potentially linearize the attention mechanism if there exist U ⋆ , V ⋆ such that f ( UV ) ≈ U ⋆ V ⋆ , as then the order of matrix multiplication can be rearranged into U ⋆ ( V ⋆ W ) which allows memory and runtime in O ( ndk ) .

In the unnormalized softmax case where f ( x ⊤ y ) = exp( x ⊤ y ) , Choromanski et al. [2021] notes that exp( x ⊤ y ) = E ζ ∼N (0 , I d ) [ exp ( ζ ⊤ x -∥ x ∥ 2 2 ) exp ( ζ ⊤ y -∥ y ∥ 2 2 )] which thus allows defining U ⋆ , V ⋆ as random feature matrices from sampled ζ 1 , . . . , ζ k . Further use of kernel properties to improve the softmax approximation have been introduced in Choromanski et al. [2022], Likhosherstov et al. [2022]. More generally, for f such that f ( x ⊤ y ) = K ( x, y ) admits a kernel representation, one may consider using variants of Bochner's theorem [Feller, 1968] to provide approximations via random Fourier features [Rahimi and Recht, 2007].

While works such as Tsai et al. [2019], Kacham et al. [2023] have experimented with linear, polynomial, exponential, and RBF kernels, so far the dominant paradigm, termed the class of Linear Transformers Katharopoulos et al. [2020], is to conveniently instead consider the reverse case, where one defines a mapping ϕ : R → R in order to define the kernel K ( x, y ) = ϕ ( x ) ⊤ ϕ ( y ) . Unfortunately, usually this does not lead to a closed-form f such that f ( x ⊤ y ) = ϕ ( x ) ⊤ ϕ ( y ) , which can lack interpretability and compatibility with classic attention mechanisms.

However, one may consider nonlinear functions f which do not admit a kernel; for example, the class of functions f ( x ) = log c ( | x | +1) for c > 0 is used in Levy and Goldberg [2014], Li et al. [2015] to compute implicit word embeddings and corresponding generative models. Within this application domain, our work thus provides answers to the question: for which classes of functions f can one efficiently compute low-rank approximations U ⋆ , V ⋆ such that f ( UV ) ≈ U ⋆ V ⋆ , especially when f ( x ⊤ y ) does not admit a kernel structure K ( x, y ) ?

2 Preliminaries

We let our implicit matrix A = f ( UV ) be n × d where U ∈ R n × r and V ∈ R r × d and n ≥ d without loss of generality, where f : R → R is a scalar function. We will also assume that d = n Ω(1) and it is often the case in empirical settings that d = n . We say that a matrix A is positive semi-definite (PSD) if it is symmetric and only has non-negative eigenvalues; a function f is a kernel function if f ( UU ⊤ ) is PSD for any matrix U . The i -th leverage score ℓ i of matrix B ∈ R n × d is equal to its sensitivity, i.e. ℓ i = sup x ∈ R d ( B T i x ) 2 ∥ Bx ∥ 2 2 = sup y ∈ colspan ( B ) y 2 i ∥ y ∥ 2 2 , where B i is the i -th row of B . The Khatri-Rao product of B ∈ R n × d is C ∈ R n × d p , where the i -th row of C is B i tensored with itself p times. We use the standard notation that ˜ O ( f ) is O ( f poly(log f )) .

For the approximate rank k LRA problem, we want to find U ′ ∈ R n × k and V ′ ∈ R k × n such that

where ∥ · ∥ denotes the Frobenius norm, unless otherwise specified and we denote [ A ] k = ˜ U ⋆ ˜ V ⋆ as some rank k matrix that minimizes the objective. Note this approximation is a relative error approximation guarantee, but can be analogously defined for additive error. For stronger lower bounds, we consider a weakened version of this problem given by outputting only a best rank k projection, specifically we want to find orthogonal W ∈ R n × k : ∥ A -AWW ⊤ ∥ ≤ (1+ ϵ ) ∥ A -[ A ] k ∥ .

Our lower bounds will rely on reductions from the conditional hardness of OVP and Max-IP, whose hardness comes from SETH. We observe, from our remarks, that some more restrictive structure can be placed on the input vector sets A,B to OVP without removing the hard instances.

Assumption 1. (Hardness of Orthogonal Vectors Problem (OVP)[Williams, 2005]). Let A = { a 1 , . . . , a n } and B = { b 1 , . . . , b d } be sets, where a i , b j ∈ { 0 , 1 } r are binary vectors for all i ∈ [ n ] = { 1 , 2 , . . . , n } , j ∈ [ d ] . Any algorithm which given any input ( A,B ) decides with constant probability if there is at least one pair of vectors a ∈ A and b ∈ B such that a T b = 0 requires ( nd ) 1 -o (1) time, provided r = ω (log n ) and r = n o (1) . Observe that the lower bounds also hold when A = B is enforced.

Remark 2. Previous work [Williams, 2005, Vassilevska Williams, 2015] has shown that Assumption 1 is true for n = d unless the Strong Exponential Time Hypothesis (SETH, Impagliazzo and Paturi [2001]) is false. Given this assumption, one can handle general n and d by a padding argument: if one could solve OVP with an algorithm A running in at most ( nd ) 1 -C time for a constant C > 0 , and without loss of generality n ≥ d , then one could solve the problem when | A | = | B | = n by splitting B into Θ( n/d ) disjoint sets each of size at most d , and solving the problem on each disjoint set in total time less than ( nd ) 1 -C · Θ( n/d ) = n 2 -C /d C ≤ n 2 -C , contradicting the assumption in the | A | = | B | = n case.

Remark 3. In Assumption 1 we can enforce more restrictive structure on the input sets A,B without making the problem easier. Specifically, we can assume that there are at most n o (1) distinct pairs with a ∈ A and b ∈ B for which a T b = 0 . Indeed, otherwise by sampling ( nd ) 2 -Ω(1) pairs at random and checking if a T b = 0 , we would solve the OVP problem in ( nd ) 1 -Ω(1) · r = ( nd ) 1 -Ω(1) time, using that r = n o (1) . This would contradict Assumption 1.

Assumption 4. (Hardness of Apx-Max-IP n,d Problem, Definition 2.1, Remark 2.2, and Lemma 4.1 of Chen and Williams [2019]). Two sets A = { a 1 , . . . , a n } and B = { b 1 , . . . , b n } are given, where a i , b i ∈ { 0 , 1 } s are binary vectors for all i ∈ [ n ] , with s = ω (log( n )) . Let m = max a ∈ A,b ∈ B a · b . Any algorithm which outputs a number ˜ m ∈ [ m/ 100 , m ] with constant probability requires ( nd ) 1 -o (1) time.

Remark 5. As in Remark 2 and Remark 3, we can reduce the general n and d case to the case n = d of previous work [Chen and Williams, 2019], and we can also enforce that there are at most n o (1) pairs a ∈ A and b ∈ B for which a · b ≥ m/ 100 , as otherwise sampling would solve the problem in less than ( nd ) 1 -o (1) time.

3 LRA Runtime Lower Bounds

We show that the complexity of low rank approximation of an entrywise transformed matrix turns out to heavily depend on the type of entrywise transformation. Based on standard complexity assumptions, we show an ( nd ) 1 -o (1) = n 1+Ω(1) time lower bound for entrywise function f ( x ) = | x | p for odd integers p in Theorem 3.1 below, while we show a weaker 2 Ω( p ) lower bound for any integer in Theorem 3.3, which becomes ( nd ) 1 -o (1) for p = Θ(log n ) . This is no accident, as we show a matching 2 O ( p ) n 1+ o (1) upper bound in Theorem 4.1. Furthermore, our lower bounds extend to functions that are approximately odd-degree polynomials of | x | , even if the degree is small. Specifically, this includes the commonly used f ( x ) = log( | x | +1) ≈ | x | .

Our proofs use variations of the Orthogonal Vectors Problem (OVP) to create two types of instances of implicit low rank approximation to create a fast O ( n 1+ o (1) ) time algorithm to solve OVP (see Algorithm 1) when given access to a low rank approximation algorithm with constant relative error and small constant additive error guarantees. Let A,B be the set of input vectors of OVP. Then, in our reduction, if there is a pair of input vectors which are orthogonal, then applying LRA , we can find either 1) a column span deviation of the low rank approximation or 2) a sparse vector in the column span of the low rank approximation. Note that we do not look at the exact column spans, but allow an additive error depending on the additive error of the output.

In the latter case, we note that our reduction structure ensures that all entries in this vector have the same magnitude. Consequently, since the column span of the output has low rank, each such entry in the support of this sparse vector corresponds to a large leverage score , and so to find the support of this unknown sparse column we can use algorithms to compute or approximate all leverage scores given the low rank factorization of the entrywise transformed matrix. This enables us to find a small superset of the support of the sparse vector, and then brute-force which pairs of entries in the superset correspond to vectors in the original OVP problem that intersect. The overall time in the reduction is negligible compared to the time to solve OVP, and therefore it must be that finding the factorization of the entrywise transformed matrix itself was expensive.

In the former case, it follows that the column spans of the output are necessarily far from each other in the two cases, and we can quickly check this and therefore solve the OVP problem. Together, it follows that the time for finding the factorization of the entrywise transformed matrix itself must have been large. While there are quantitative differences in the two cases of odd and even integers p , the proofs both follow this strategy.

Algorithm 1 OVP to LRA Reduction

Input: Sets A = { a 1 , . . . , a n ∈ { 0 , 1 } s } and B = { b 1 , . . . , b n ∈ { 0 , 1 } s } ⊤

Output: YES if there exists a b = 0 and NO otherwise.

1: Choose c ∈ {-1 , 1 } n uniformly at random

2: Let U = [ A | c ] ∈ R n × ( s +1) and V ⊤ = [ B | c ] ∈ R n × ( s +1) and r = s +1 .

3: Let W = LRA ( U , V , f = | x | p ) for constant relative error and α additive error

4: Compute U ′′ = U ⊗ ... ⊗ U ∈ R n × r p , U tensored with itself p times and similarly for V ′′ .

5: Let r 2 i = ∥ U ′′ V ′′ e i ∥ 2 2 -∥ U ′′ V ′′ WW ⊤ e i ∥ 2 2 for i -th column ▷ Calculate column distances of U ′′ V ′′ to column span of W .

6: If any of r 2 i > 1 . 01 α , output YES

▷α is any upper bound on additive error of LRA

7: Compute 1 / 2 -approximate leverage scores ℓ i of U ′′ and let S = { i ∈ [ n ] | ℓ i ≥ 1 / (100 n o (1) ) } ▷ Finds the representative subset and the threshold is given by the OVP assumption.

8: Compute all dot products between a i and B for all i ∈ S . Output YES if there exists a pair such that a ⊤ i b j = 0 and NO otherwise

3.1 General Functions of | x | p for odd integers p

In the following theorem we will also allow for a tiny amount of additive error, as this will be useful for our later lower bound applications where we approximate other functions using a Taylor series and reduce from the following theorem.

Theorem 3.1. Suppose n ∈ N , f ( x ) = | x | p for an odd integer constant p = O (log( n )) , and r = O (log( n )) . There is a positive integer k = n o (1) , such that for any approximation factor ∆ ≥ 1 and any constant α < 2 , any possibly randomized, algorithm which given any input U ∈ R n × r and V ∈ R r × d outputs W ∈ R n × k satisfying

with constant probability, requires ( nd ) 1 -o (1) time, under Assumption 1. Further, this holds even if U = V T . Here [ f ( U · V )] k denotes the best rankk approximation to f ( U · V ) in the Frobenius norm.

Proof. Suppose A and B are input sets to the OVP problem of Assumption 1 with parameter r = s +1 , where s is the dimension of the points. We let the rows of U be the points in A , but we append one additional dimension, represented as column vector c to U that is chosen uniformly at random in {-1 , 1 } n . Similarly, the columns of V correspond to the points in B , but we append one additional row to V , which is equal to c T . Observe that if A = B , then necessarily U = V ⊤ .

Let Case 1 be when there is no a ∈ A and b ∈ B with a T b = 0 , and Case 2 be when there is an a ∈ A and a b ∈ B with a T b = 0 , and we are deciding whether we are in Case 2. We claim that our algorithm (Algorithm 1) will always output NO when we are in Case 1 and will output YES with constant probability when we are in Case 2. This clearly suffices to solve OVP with constant probability.

Notice that if there is no a ∈ A and b ∈ B for which a T b = 0 , then for all a ∈ A and b ∈ B , we have a T b ≥ 1 . Consequently with probability 1 over the choice of column vector c , for all i ∈ [ n ] and j ∈ [ d ] , U i V j ≥ 0 , where U i is the i -th row of U and V j is the j -th column of V . Consequently, f ( U i V j ) = ( U i V j ) p . Thus, f ( U · V ) = U ′′ V ′′ , where each row of U ′′ ∈ R n × r p is the Khatri-Rao product of itself p times, and each column of V ′′ ∈ R r p × d is the Khatri-Rao product of itself p times.

Observe that the rank of f ( U · V ) is at most rank ( U ′′ ) + n o (1) ≤ ( s +1) p + n o (1) since by Remark 3 we have at most n o (1) dot product pairs that are zero, implying that f ( U · V ) can be obtained from U ′′ · V ′′ by changing at most n o (1) entries. It follows that there is a value k ≤ n o (1) +( s +1) p such that, in both cases, for any multiplicative approximation factor ∆ ≥ 1 , necessarily the output W satisfies ∥ f ( U · V ) WW ⊤ -f ( U · V ) ∥ 2 F ≤ α with constant probability.

Next, we compare the column span of W to that of U ′′ · V ′′ . In Case 1, there is no pairwise dot product that is zero, so we have that f ( U · V ) = U ′′ V ′′ . Therefore, each column of U ′′ · V ′′ has squared distance at most α to the column span of W .

Note that we can compute all squared distances of U ′′ V ′′ to the column span of W in O ( n 1+ o (1) ) time. This is so that we can detect whether we are in case 1, where U ′′ V ′′ has at squared distance at most α to the column span of W , or we are in case 2. Specifically, we compute the squared distance by the Pythagorean theorem and since W is orthogonal:

and for each i , we can compute this in n o (1) time, given the matrices we have precomputed in the previous paragraph. Thus, we can compute all squared distances up to additive 1 / poly( n ) in n 1+ o (1) time. If we see that any squared distance is larger than α +1 / poly( n ) , we know we are in Case 2.

Therefore, we may assume in what follows that each column of U ′′ · V ′′ has squared distance at most α +1 / poly( n ) from the column span of W . In this case we can also determine in n 1+ o (1) time, with constant probability, whether we are in Case 2. To do so, we apply a leverage score approximation algorithm [Clarkson and Woodruff, 2013] to find a small representative subset A ′ ⊂ A such that | A ′ | = n o (1) and if there exists a ∈ A,b ∈ B such that a ⊤ b = 0 , then a ∈ A ′ . Specifically, suppose there is an a ∈ A and a b ∈ B for which a T b = 0 , and suppose U i extends a by one coordinate and V j extends b by one coordinate. Then with probability at least 1 / 2 , U T i V j = -1 , and so f ( U T i V j ) = 1 . Let us condition on this event. Then we have the following claim in this case.

Claim 6. Let a ∈ A and b ∈ B be such that a T b = 0 , and the corresponding row U i and column V j satisfy U ⊤ i V j = -1 . Also, suppose each column of U ′ · V ′ has squared distance at most α +1 / poly( n ) from the column span of U ′′ . Then, in n 1+ o (1) time, we can find a representative subset A ′ ⊂ A of size n o (1) such that a ∈ A ′ with high probability.

Proof. (of Claim) First note that by Remark 3, there can be at most n o (1) coordinates in the j -th column of f ( U · V ) which differ from their corresponding coordinate value in the j -th column of U ′′ V ′′ , since each such difference corresponds to a pair of points a ∈ A and b ∈ B in the OVP problem for which a T b = 0 . Hence, since f ( U · V ) has squared distance at most α +1 / poly( n ) from the column span of W , then consider the following vectors by taking the difference of the j -th column of f ( U · V ) and the j -th column of U ′′ · V ′′ , we would have vectors v denoting the difference vector and e being the residual vector of v projected on the column span of U ′′ , with the following properties:

v contains at most n o (1) values, each of value equal to 2 ,

∥ e ∥ ≤ α +1 / poly( n )

v + e is in the column span of U ′′ .

Let S ⊂ [ n ] be the set of of indices i for which ( v + e ) 2 i ∥ v + e ∥ 2 2 ≥ 1 /n o (1) . Since α +1 / poly( n ) is at most a constant strictly less than 2 , it follows that for each i in the support of v , since v i = 2 , we have that ( v + e ) 2 i ∥ v + e ∥ 2 2 ≥ 1 /n o (1) . Since also ∥ v + e ∥ 2 2 ≤ n o (1) , the support of v is included in S and | S | ≤ n o (1) .

Recall the definition of leverage scores ℓ i from Section 2 and that the sum of the n leverage scores of B ∈ R n × t is exactly equal to the rank of B , which is at most t . Since v + e is in the column span of U ′′ , the i -th leverage score of U ′′ satisfies ℓ i ( U ′′ ) ≥ ( v + e ) 2 i / ∥ v + e ∥ 2 2 = Ω(1 /n o (1) ) .

Consequently, each coordinate i in S satisfies ℓ i ( U ′′ ) = Ω(1 /n o (1) ) . It is known [Clarkson and Woodruff, 2013] how, given a matrix B , in O ( nt log n )+ t O (1) time, one can compute a list ℓ ′ 1 , . . . , ℓ ′ n with ℓ ′ i = Θ( ℓ i ) for all i with probability 1 -1 /n 100 . Consequently, using that rank ( U ′′ ) ≤ r p , given U ′′ one can find a superset T containing S for which | T | = O ( r p n o (1) ) in n 1+ o (1) time, where recall r = s +1 and s = n o (1) and p is constant. Note that our bound on | T | follows since we just need to keep the leverage scores that are Ω(1 /n o (1) ) and the sum of all leverage scores is at most r p = n o (1) , recalling r = s +1 and s = n o (1) .

Continuing the proof of Theorem 3.1, by our claim, when we are in Case 2, with at least constant probability, we find in time O ( n 1+ o (1) ) a representative subset A ′ of size O ( n o (1) ) . Given our subset A ′ , one can then compute all pairs of dot products between the points in A ′ and the points in B in O ( n o (1) ds ) time, and since we may assume n ≥ d without loss of generality, we can compute all such pairs in n 1+ o (1) time. Lastly, if no such a, b exist, when we are in Case 1, this algorithm cannot err. Therefore, we have an algorithm that can solve OVP , which then violates Assumption 1, using that ( nd ) 1 -o (1) = n 1+Ω(1) .

Our lower bound techniques also apply to a number of important function f that do not have the form of | x | p for an integer p . We show that our lower bounds for LRA in fact holds for any function g ( x ) = f ( | x | ) , where f ( x ) is a function that admits a Taylor expansion with a dominant term of x p , for odd p , around 0 . Of particular interest in natural language processing [Liang et al., 2020, Jiang et al., 2021] is the function g ( x ) = log(1 + | x | ) , where satisfies g ( x ) = f ( | x | ) , where f ( x ) = x + O ( | x | 2 ) and thus we can appeal to our lower bound with p = 1 .

Theorem 3.2. Suppose n ∈ N , r = O (log( n )) , and g ( x ) = f ( | x | ) , where f admits a Taylor expansion f ( x ) = c p x p + O ( | x | p +1 ) for odd p = O (log( n )) . There is a positive integer k = n o (1) , such that for any approximation factor ∆ ≥ 1 , any, possibly randomized, algorithm which given inputs U ∈ R n × r and V ∈ R r × d outputs W ∈ R n × k satisfying

with constant probability, requires ( nd ) 1 -o (1) time, under Assumption 1. Further, this holds even if U = V ⊤ .

3.2 Polynomials of All Integers p

We now give the proof for p -degree polynomials for all integers p , which is weaker than Theorem 3.1 for odd integers, but this is the first such lower bound for even integers. The weaker bound cannot be substantially improved since in this setting f ( x ) = x p , so f ( UV ) is in fact a kernel and matrix vector multiplication can be performed in O ( nr p ) time. For intuition why our lower bound techniques do not extend, recall that our previous reduction forces the absolute value operation to essentially alter entries of ( UV ) p but only at entries of the original with zero dot product. Therefore, we can write as a sum of a low rank matrix (from tensor product) and a sparse matrix, whose sparse entries now represent the OVP pairs. However, when p is even, the absolute value operation leaves the entries unchanged and does not induce the additional sparse matrix. Therefore, our lower bounds for this setting rely on a slightly different variant of OVP, specifically quadratic lower bounds for finding the maximum dot product (Assumption 4).

Intuitively, our new reduction is as follows: Let OPT be the maximum inner product and by assumption, consider this small set of large inner product pairs, which represents a small number of entries in that are large in magnitude so that when you apply a threshold at OPT / 100 , the resulting matrix is sparse. Since f ( x ) = x p amplifies the magnitude differences, it follows that ( UV ) p is much closer relatively to an approximately sparse, and therefore low rank, matrix. Therefore an approximate low rank approximation (LRA) algorithm can recover this sparse low rank matrix well enough so that the span of the approximate matrix can be used, via leverage score computations, to solve the APX-Max-IP problem.

Theorem 3.3. Let U ∈ R n × r and V ∈ R r × d be given with r = O (log( n )) . Suppose f ( x ) = x p for an integer p ≥ 1 . There is a positive integer k = n o (1) , such that for any approximation factor ∆ ≥ 1 , any, possibly randomized, algorithm which outputs W ∈ R n × k satisfying ∥ f ( U · V ) WW ⊤ -f ( U · V ) ∥ 2 F ≤ ∆ · ∥ [ f ( U · V )] k -f ( U · V ) ∥ 2 F , with constant probability for any constant ∆ > 1 , requires min(( nd ) 1 -o (1) , 2 Ω( p ) ) time, under Assumption 4.

The lower bound for general functions f ( x ) in Theorem 3.2 with odd-degree dominating term in its Taylor expansion works because we can scale down the entries in our input matrix so that our relative-error approximation guarantee is still preserved but the function is largely approximated by only the leading polynomial term. By a combination of the same taylor expansion argument with a slight more general version of Theorem 3.3 that handles small additive error, we note that we can extend our results to when p is even.

We note that our lower bounds do not generalize to the case when U = V because as mentioned in our intuitive introduction, our reduction hinges on the sparse low-rank structure of ( UV ) p . However, when U = V , the sparsity structure is broken as the diagonal of the matrix is now larger than the maximum inner product of two different vectors, and this destroys the low rank structure. In some sense, the diagonal of our matrix is forced to include the dot product of u i with itself and this shifts the entire matrix by a large multiple of the identity, crucially removing the sparse + low-rank structure that we exploited in our lower bound argument before. Indeed, as we remark below, the resulting positive semidefinite (PSD) structure allows us to derive a fast LRA algorithm and our lower bound no longer holds in this setting.

̸

Remark 7. Recall that when U = V ⊤ , there is an nr ( k/ϵ ) ω -1 subquadratic time algorithm when k = O ( n o (1) ) , for (1 + ϵ ) -relative error approximation, but reducing the U = V ⊤ case to the U = V ⊤ case requires blowing up k to k + r p . Therefore, our lower bounds imply that the U = V ⊤ case is strictly more difficult in terms of runtime in certain settings.

4 LRA Algorithms from Matrix Vector Products

In this section, we can show upper bounds for low rank approximation of entrywise transformed products that are O ( n 1+ o (1) ) when f ( x ) represents a kernel matrix but U = V ⊤ . Specifically, we focus on polynomial functions of the form f ( x ) = x p and demonstrate that our lower bounds are tight for relative error LRA. Note that we have shown that low rank approximation guarantees for PSD matrices when U = V ⊤ cannot translate to the case when U = V ⊤ . Still, we demonstrate that relative error low rank approximation is possible in n 1+ o (1) time for polynomial kernels with even degree, although there will be an exponential dependence on p for relative error low rank approximation. Our relative error algorithms are relatively standard and rely on low rank projections by using matrix vector products to perform randomized sketching to reduce our row or column dimension to O ( poly ( k/ϵ )) [Woodruff, 2014].

̸

Theorem 4.1. Let U ∈ R n × r and V ∈ R r × d . Suppose f ( x ) = x p for an even integer p ≥ 1 and k < r p . For any approximation factor ϵ > 0 , there is an algorithm that outputs U ′ ∈ R n × k and V ′ ∈ R k × d satisfying ∥ U ′ · V ′ -f ( U · V ) ∥ 2 F ≤ (1 + ϵ ) · ∥ [ f ( U · V )] k -f ( U · V ) ∥ 2 F with constant probability with runtime O (( n + d ) r p k/ϵ 3 + poly ( r p /ϵ )) .

Proof. Note that f ( U i V j ) = ( U i V j ) p . Thus, we may rewrite f ( U · V ) = U ′′ V ′′ , where each row of U ′′ ∈ R n × r p is the Khatri-Rao product of itself p times, and each column of V ′′ ∈ R r p × d is the Khatri-Rao product of itself p times. Note that the rank of U ′′ V ′′ is at most d p .

Let S be a random Gaussian sketching matrix with O ( k/ϵ ) rows, so we know that these matrices satisfy the ( √ ϵ/k, 9 / 10 , l ) -JL property [Woodruff, 2014]. Furthermore, let R be a random Gaussian matrix with O (min( k/ϵ 3 , r p /ϵ 2 )) columns, so we know that it is a (1+ O ( ϵ )) ℓ 2 subspace embedding of the row space of SU ′′ . Then, by Theorem 47 of Woodruff [2014], the following is true with constant probability

Finally we bound the runtime of computing this product. Note that we may compute SU ′′ and V ′′ R in nr p · ( k/ϵ +min( k/ϵ 3 , r p /ϵ 2 )) time. Then, note that the remaining products can be computed in

̸

̸

poly ( kr p /ϵ ) = poly ( r p /ϵ ) time. And lastly, the rank k approximation follows from solving for the low rank approximation on the restricted subspace SU ′′ V ′′ R of rank and letting Z = [ U ′′ V ′′ RU ] k , where U is the orthonormal basis such that UU ⊤ is the projection onto the row space of SAR.

We also provide an additive error guarantee on the low rank approximation guarantees that depends polynomially on p , as opposed to the tight exponential dependence in the relative error setting. This implies that additive error LRA is strictly easier and can be accomplished in subquadratic time when p = Ω(log( n )) . Note that lower bounds for additive error are not emphasized in this paper as the upper bound is already quite competitive and simply outputting requires Ω( n ∗ poly( p )) .

Our guarantees follow from tensor-based sketching techniques that are applied along each tensor dimension to remove the exponential dependence on p . We note that this results in an additive error term that is on the order of the p -norms of U, V . This term is related to the absolute error guarantees of [Han et al., 2020], which also depends on the product of the p -norms of Euclidean norms of the rows of U, V .

Algorithm 2 TensorSketch LRA

Input:

Matrices U ∈ R , V ∈ R , p > 0 even integer n × r r × d

Output: rank k approximation of f ( UV ) , where f ( x ) = | x | p

-2

1: Let T be a TensorSketch with m = O ( pϵ ) rows ▷ See Ahle et al. [2020]

2: Compute U ′′′ = U ′′ T ⊤ and V ′′′ = TV ′′ , where U ′′ ∈ R n × r p , V ′′ ∈ R r p × d are U , V tensored themselves p times.

3: Let S , R be random Gaussian matrices with O ( k/ϵ ) rows and O ( p/ϵ 4 ) columns respectively

′′′ ′′′ . ▷ This implies

4: Compute an orthonormal basis P of row span of SU V R PP ⊤ = ( SU ′′′ V ′′′ R ) + ( SU ′′′ V ′′′ R )

5: Compute the rank k decomposition: U ′ V ′ = [ U ′′′ V ′′′ RP ] k . ▷ [ M ] k is the best rank k approximation to M

6: Output U ′ , V ′ P ⊤ ( SU ′′′ V ′′′ R ) + SU ′′′ V ′′′

Theorem 4.2. Let U ∈ R n × r and V ∈ R r × d . Suppose f ( x ) = x p for an even integer p ≥ 1 and k < r p . For any approximation factor ϵ > 0 , there is an algorithm (Algorithm 2) that outputs U ′ ∈ R n × k and V ′ ∈ R k × d satisfying ∥ U ′ · V ′ -f ( U · V ) ∥ 2 F ≤ (1 + ϵ ) · ∥ [ f ( U · V )] k -f ( U · V ) ∥ 2 F + ϵ 2 L 2 with constant probability with runtime O (( n + d ) · poly ( p, r, k, 1 /ϵ )) , where the additive term is given by L 2 = ( ∑ n i =1 ∥ U i ∥ 2 p 2 )( ∑ d i =1 ∥ V i ∥ 2 p 2 ) = ∥ U ∥ 2 p 2 p, 2 ∥ V ⊤ ∥ 2 p 2 p, 2 .

4.1 Lower Bounds on Matrix Multiplication

Our low rank approximation algorithms use a matrix vector multiplication subroutine f ( UV ) z i for O ( poly ( k/ϵ )) different vectors z i as their main dimensionality reduction technique for subquadratic time upper bounds. This can be used to directly translate lower bounds for low rank approximation to implicit matrix vector multiplication for a wide range of scalar functions that extends previous work beyond the exponential function. We emphasize that these matrix-vector product lower bounds also imply that our LRA lower bounds are non-trivial and significantly strengthen those provided by previous works.

Theorem 4.3 (LRA reduces to Matrix Vector Products) . Let U, V, f be as in Theorem 3.2. Then, any possibly randomized algorithm that for any vector z ∈ R n , outputs f ( UV ) z up to 1 / poly(n) entrywise error, with constant probability, requires Ω(( nd ) 1 -o (1) ) time, under Assumption 1. This holds even when U = V ⊤ .

References

Thomas D Ahle, Michael Kapralov, Jakob BT Knudsen, Rasmus Pagh, Ameya Velingker, David P Woodruff, and Amir Zandieh. Oblivious sketching of high-degree polynomial kernels. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 141-160. SIAM, 2020.

Josh Alman and Zhao Song. Fast attention requires bounded entries. arXiv preprint arXiv:2302.13214 , 2023.

Josh Alman, Timothy Chu, Aaron Schild, and Zhao Song. Algorithms and hardness for linear algebra on geometric graphs. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 541-552. IEEE, 2020.

Arturs Backurs, Piotr Indyk, and Ludwig Schmidt. On the fine-grained complexity of empirical risk minimization: Kernel methods and neural networks. In Advances in Neural Information Processing Systems , 2017.

Ainesh Bakshi, Nadiia Chepurko, and David P Woodruff. Robust and sample optimal algorithms for PSD low rank approximation. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 506-516. IEEE, 2020.

Lijie Chen and Ryan Williams. An equivalence class for orthogonal vectors. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 21-40. SIAM, 2019.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tam´ as Sarl´ os, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with Performers. In 9th International Conference on Learning Representations, ICLR 2021 , 2021.

Krzysztof Marcin Choromanski, Han Lin, Haoxian Chen, Arijit Sehanobish, Yuanzhe Ma, Deepali Jain, Jake Varley, Andy Zeng, Michael S. Ryoo, Valerii Likhosherstov, Dmitry Kalashnikov, Vikas Sindhwani, and Adrian Weller. Hybrid random features. In The Tenth International Conference on Learning Representations, ICLR 2022 , 2022.

Kenneth L. Clarkson and David P. Woodruff. Low rank approximation and regression in input sparsity time. In Symposium on Theory of Computing Conference, STOC 2013 , pages 81-90. ACM, 2013.

William Feller. An Introduction to Probability Theory and Its Applications , volume 1. Wiley, January 1968. ISBN 0471257087.

Insu Han, Haim Avron, and Jinwoo Shin. Polynomial tensor sketch for element-wise function of low-rank matrix. In International Conference on Machine Learning , pages 3984-3993. PMLR, 2020.

Russell Impagliazzo and Ramamohan Paturi. On the complexity of k-SAT. Journal of Computer and System Sciences , 62(2):367-375, 2001.

Yifei Jiang, Yi Li, Yiming Sun, Jiaxin Wang, and David Woodruff. Single pass entrywise-transformed low rank approximation. In International Conference on Machine Learning (ICML) , pages 49824991. PMLR, 2021.

Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketches for polynomial kernels. arXiv preprint arXiv:2310.01655 , 2023.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5156-5165. PMLR, 2020. URL http://proceedings.mlr.press/v119/katharopoulos20a.html .

Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention. In International Conference on Algorithmic Learning Theory (ALT) , 2023.

Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems , 2014.

Shaohua Li, Jun Zhu, and Chunyan Miao. A generative word embedding model and its low rank positive semidefinite solution. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 1599-1609, 2015.

Yingyu Liang, Zhao Song, Mengdi Wang, Lin Yang, and Xin Yang. Sketching transformed matrices with applications to natural language processing. In International Conference on Artificial Intelligence and Statistics , pages 467-481. PMLR, 2020.

Valerii Likhosherstov, Krzysztof M Choromanski, Kumar Avinava Dubey, Frederick Liu, Tamas Sarlos, and Adrian Weller. Chefs' random tables: Non-trigonometric random features. In Advances in Neural Information Processing Systems , pages 34559-34573, 2022.

Daniel Lokshtanov, D´ aniel Marx, Saket Saurabh, et al. Lower bounds based on the exponential time hypothesis. Bulletin of EATCS , 3(105), 2013.

Cameron Musco and Christopher Musco. Recursive sampling for the Nystrom method. In Advances in Neural Information Processing Systems , 2017.

Cameron Musco and David P Woodruff. Sublinear time low-rank approximation of positive semidefinite matrices. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 672-683. IEEE, 2017.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems , 2007.

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6):1-28, 2022.

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 4343-4352. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1443. URL https://doi.org/10.18653/v1/D19-1443 .

Virginia Vassilevska Williams. Hardness of easy problems: Basing hardness on popular conjectures such as the strong exponential time hypothesis (invited talk). In 10th International Symposium on Parameterized and Exact Computation (IPEC 2015) . Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 2017 , pages 5998-6008, 2017.

Ryan Williams. A new algorithm for optimal 2-constraint satisfaction and its implications. Theor. Comput. Sci. , 348(2-3):357-365, 2005.

David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science , 10(1-2):1-157, 2014.

David P Woodruff and Peilin Zhong. Distributed low rank approximation of implicit functions of a matrix. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE) , pages 847-858. IEEE, 2016.

A Missing Lower Bound Proofs

Proof of Theorem 3.2. Let U, V T ∈ {-1 , 0 , 1 } n × r be an instance of entrywise transformed low rank approximation with g . WLOG, by scaling our solution, we can let c p = 1 . Let B = poly( n ) be sufficiently large, and let ˜ U = U/ √ B and ˜ V = V/ √ B . For each i, j ∈ [ n ] , for B sufficiently large and using that r = n o (1) ,

where h ( x ) = | x | p . Hence, if ˜ U ′ , ˜ V ′ are such that

then by the triangle inequality,

Setting U ′ = √ B · ˜ U ′ and V ′ = √ B · ˜ V ′ and scaling both sides by B , we have

where in the second inequality we used that [ h ( ˜ U · ˜ V )] k has rank k whereas [ g ( ˜ U · ˜ V )] k is the best rankk approximation to g ( ˜ U · ˜ V ) in Frobenius norm. Note that the third line is triangle inequality and the fourth line follows our entrywise approximation bounds.

Now by AM-GM, using that a ≤ b + c implies that a 2 ≤ 2 b 2 +2 c 2 for a, b, c ≥ 0 , we have

Thus, U ′ · V ′ is a rankk approximation to h ( U · V ) with additive error 1 poly( n ) , by setting B to be large enough, and relative error 2∆ . The total time to find U ′ and V ′ is the same as the time to solve entrywise transformed low rank approximation with respect to the function g on inputs ˜ U and ˜ V , and thus by Theorem 3.1 is at least ( nd ) 1 -o (1) . Note that applying the case U = V ⊤ in Theorem 3.1 establishes this theorem when ˜ U = ˜ V ⊤ .

Proof of Theorem 3.3. Suppose A and B are the input sets to the Apx-Max-IP n,d Problem of Assumption 4 with parameter s = r . We let the rows of U be the points in A and we let the columns of V be the points in B . As in the proof of Theorem 3.1, we have f ( U T i V j ) = ( U T i V j ) p so f ( U · V ) = U ′′ · V ′′ . By Remark 5, there are at most n o (1) entries of f ( U · V ) that are at least ( m/ 100) p .

Hence, if we set k to be an appropriate value in n o (1) , then

since one possible rankk approximation is just to zero out the at most n o (1) entries of f ( U · V ) that are at least ( m/ 100) p . By the correctness guarantee, with constant probability,

We may assume WLOG that p ≥ C log n for the moment, for a sufficiently large constant C > 0 since a trivial lower bound for outputting an LRA is Ω( n ) , so the inclusion of the 2 Ω( p ) term in the lower bound allows us to make this assumption. Therefore, we conclude that f ( U · V ) is in the column span of W up to 1 / poly ( n ) error. Consequently, consider the maximum entry m , which we can assume is at least 1 , as otherwise all points in A would have disjoint support from those in B but this can be verified in O ( nr ) = n 1+ o (1) time, contradicting the n 1+Ω(1) lower bound for d = n Ω(1) in Assumption 4.

It follows that if the maximum m occurs in the ( i, j ) -th entry of f ( U · V ) = U ′′ V ′′ , then there exists a vector in the column span of U ′′ for which the i -th entry is [ m p (1 -poly ( n )) , m p (1 + poly ( n ))] and all other entries are in the range [ -m p / poly ( n ) , m p / poly ( n )] since (1 / 100) p = 1 / poly ( n ) . Consequently, since the column span of W is close to the column span of U ′′ up to 1 / poly ( n ) error, we see that the column span of W also contains these almost-sparse vectors that have n o (1) large entries. As argued before similarly in Theorem 3.1, the rows of these large dot products forces the row leverage scores of W to be Ω(1) . Since W has rank k = n o (1) , in n 1+ o (1) time we can find the set S of n o (1) row leverage scores of W that are Ω(1) . We can then explicitly compute all dot products between pairs of vectors in A ∩ S and B in n 1+ o (1) total time, at which point we can output the maximum dot product, which includes m . Therefore by Assumption 4, the total time to find W is at least ( nd ) 1 -o (1) .

Proof of Theorem 4.3. This proof follows by reducing low rank approximation to matrix multiplication by the same algorithm as in Theorem 4.1. Specifically, let us set k = n o (1) and n = d and note that our low rank approximation algorithm's runtime is dominated by computing S · f ( UV ) and f ( UV ) · R , where S, R are matrices with poly ( k/ϵ ) rows and columns, respectively. In fact, those operations are the only terms that incur a polynomial dependence on n .

Therefore, suppose there exists such a matrix multiplication algorithm that takes time O ( n 2 -c ) for some c . Then this would directly imply that computing both S · f ( UV ) , f ( UV ) · R takes O ( n 2 -c ) time as [ Sf ( UV )] ⊤ = f ( V ⊤ U ⊤ ) S ⊤ . By our guarantees in Theorem 47 of Woodruff [2014], this implies a constant relative error LRA in time O ( n 2 -c ) , which contradicts Theorem 3.2.

B Missing Upper Bound Proofs

Proof of Theorem 4.2. Again, f ( U i V j ) = ( U i V j ) p so we may rewrite f ( U · V ) = U ′′ V ′′ , where each row of U ′′ ∈ R n × r p is the Khatri-Rao product of itself p times, and each column of V ′′ ∈ R r p × d is the Khatri-Rao product of itself p times.

Using Theorem 1 of Ahle et al. [2020], we see that with probability 0 . 9 , our approximate matrix product guarantees hold for U ′′ , V ′′ such that if Π is an m × r p TensorSRHT matrix with m = Θ( p/ϵ 2 ) , then

Therefore, we can use the approximate low rank sketches again to approximately solve:

Specifically, let S be a random Gaussian sketching matrix with O ( k/ϵ ) rows. We know that these matrices satisfy the ( √ ϵ/k, 9 / 10 , l ) -JL property Woodruff [2014]. Furthermore, let R be a random

Gaussian matrix with O ( m/ϵ 2 ) columns, so we know that it is a (1 + O ( ϵ )) ℓ 2 subspace embedding of the row space of SU ′′ Π ⊤ , which has rank at most m . Let U ′′′ = U ′′ Π ⊤ and let Π V ′′ = V ′′′ . Then, by Theorem 47 of Woodruff [2014], the following is true with constant probability

Finally we bound the runtime of computing this product. Note that we may compute U ′′′ and V ′′′ in time O ( np ( m + r )) time. Then, computing SU ′′′ , V ′′′ R can be done in nm · ( k/ϵ + m/ϵ 2 ) time. Lastly, the remaining products can be computed in poly ( kp/ϵ ) and the final rank k decomposition can be computed and we can find U ′ , V ′ such that

Now, let U ⋆ , V ⋆ be the optimal rank k decomposition of the original problem of f ( UV ) , then by the guarantees of approximate matrix product and the triangle inequality, ∥ U ⋆ V ⋆ -U ′′ V ′′ ∥ F ≥ ∥ U ⋆ V ⋆ -U ′′ Π ⊤ Π V ′′ ∥ F -ϵ ∥ U ′′ ∥ F ∥ V ′′ ∥ F .

Therefore, we conclude that

We end by rewriting ∥ U ′′ ∥ 2 F = Tr( U ′′ U ′′⊤ ) = ∑ n i =1 ( U ⊤ i U i ) p = ∑ i ∥ U i ∥ 2 p 2 and similarly for ∥ V ′′ ∥ 2 F and then applying AM-GM and noting that (1 + ϵ ) 2 = 1 + O ( ϵ ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of computing a low-rank approximation (LRA) for $f(UV)$, where $U\\in \\mathbb{R}^{n\\times r}$, $V\\in \\mathbb{R}^{r\\times n}$, and $f:\\mathbb{R}\\rightarrow \\mathbb{R}$ applies to each entry of $UV$. This problem has very important applications in deep learning, in particular for Transformers and natural language processing. The main results of this paper consist of two parts:\n\nFor lower bounds, they show that we cannot get subquadratic relative error LRA generally when either 1) $U\\ne V^\\top$ even for PSD kernel functions or 2) for $f$ that are approximately polynomials of $|x|$ even for a constant degree. \n\nFor upper bounds, they show that for $f(x) = x^p$, there is an $O(n\\cdot poly(r^p, k, 1/\\epsilon))$-time algorithm for relative error LRA and $O(n \\cdot poly(p, k, 1/\\epsilon))$-time for additive error LRA. \n\nFurthermore, their techniques can be generalized to obtain lower bounds for the implicit matrix-vector multiplications of the form $f(UV)z$.\n\nSTRENGTHS:\nThis paper significantly advances our understanding of LRA with very solid results. Prior to this work, most of the research focuses on the symmetric setting where $U=V^\\top$. This paper considers the more general, asymmetric setting. For relative error LRA and $f(x)=x^p$, their lower bound almost matches their upper bound in terms of the dependence on $p$. On the other hand, they also make progress in the symmetric setting. Their lower bound in the symmetric setting applies to a larger family of the function $f$ compared to previous results. Technically, they provide clean and novel reductions from the orthogonal vector problem (OV) to LRA via the fast computation of leverage scores. The reductions in this paper will be useful in proving new lower bounds for other numerical linear algebra problems. Additionally, this paper provides a clear review of many related works about LRA.\n\nWEAKNESSES:\nThe algorithms proposed in this paper is quite straightforward given prior results  [Woodruff, 2014] and [Ahle et al., 2020]. Also, the lower bound for LRA with an even $p$ is weak. And the lower bounds do not depend on $r$ and $k$, the dimensions of the input matrices and output matrices, respectively. Moreover, there are some typos in the proofs that affects reading this paper.\n\nQUESTIONS:\n1. Line 157: define $B_i$.\n\n2. Do the lower bounds for OV (Assumption 1) and Max-IP (Assumption 4) hold for randomized algorithms? They may be based on (randomized) SETH.\n\n3. Line 262: $s\\leq r^p$.$s$ has already been used to represent the number of vectors in OV. I guess the $s$ here is different?\n\n4. Line 265: the derivation of the below equation via the Pythagorean theorem is unclear to me. And how does it imply that when we are in case 2, there is a column of $U’V’$ that is $\\alpha+1/poly(n)$ far from the column span of $U’’$?\n\n5. Line 287: how do you construct the vectors $v$ and $e$?\n\n6. Line 306: where does the $n^{o(1)}dr^ps$ come from? Shouldn’t it be $n^{o(1)}ds$?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
The authors discuss a setting where one observes a scalar transformation of an r-rank matrix, f(UV) and wants to find the best k-rank approximation of it.

It is known that under various assumptions, if f(UU^t) is PSD then the problem is solvable in nearly linear time. It is also known that for various kernels f (eg Gaussian kernel), one needs quadratic time.

The authors discuss the cases where f are simple monomials, f(x)=x^p. They show a min(2^p,n^2) time lower bound based on SETH.  They also show a matching upper bound, among other results.

STRENGTHS:
The paper seems novel and interesting. The connection with SETH is non-trivial and intruiguing.

WEAKNESSES:
I am failing to see the importance of choosing the x^p activation function (besides the log(1+|x|) example). Could the authors elaborate more on it?

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of low rank approximation of the form f(U.V) where the matrices U,V $\\in \\mathbb{R}^{n \\times r}$ and $r=O(\\log n)$ and $f(.)$ is applied entrywise. The main contributions of the paper are the following:\n\n1) The paper presents a $\\Omega(n^{2-o(1)})$ time lower bound for getting relative error rank $k$ approximation to $f(U.V)$ for $k=n^{o(1)}$ and $f(x)$ admits a Taylor series expansion of the form $f(x)=x^p+O(|x|^{p+1})$ when $p$ is odd and $p=O(\\log n)$. This lower bound holds for any $U, V$. Another $\\min(n^{2-o(1)},2^{\\Omega(p)})$ time lower bound, which only holds when $U \\neq V^T$, is presented for the case when $f(x)=x^p$ for any general integer $p \\geq 1$. The first lower bound proof for odd $p$ follows by a reduction of the Orthogonal Vectors Problem to the Low-Rank Approximation (LRA) problem of $f(U.V)$. The proof then proceeds by lower bounding the leverage score of rows corresponding to non-zero entries of a flat sparse vector in the column span of the low-rank approximation of $f(U.V)$. The second lower bound proof for a general $p$ follows by a reduction of the Approximate Maximum Inner Product problem respectively to the LRA problem.\n\n2) The paper presents a $O(n . poly(2^p,k,\\frac{1}{\\epsilon}))$ time algorithm for getting a $1+\\epsilon$ relative error low-rank approximation to $f(U.V)$ mathcing the lower bound. The algorithm uses random Gaussian sketching of the Khatri-Rao products of U and V followed by a low-rank approximation of the smaller sketches. The paper also presents a $O(n, poly(k,p, \\frac{1}{\\epsilon}))$ time algorithm for getting a low-ran approximation with additive $\\epsilon . L$ guarantee where $L$ depends on the $p$-norms of $U,V$. Thi algorithm proceeds by doing a tensor sketch of the Kahtri-Rao products of $U$ and $V$.\n\n3) Finally, the paper presents a lower bound of $\\Omega(n^{2-o(1)})$ on matrix vector products of the form $f(UV)z$ for a vector $z \\in \\mathbb{R}^n$ using the fact that the algorithms use a matrix vector product subroutine.\n\nSTRENGTHS:\n1) Previous subquadratic time algorithms for relative error low-rank approximation for $f(U.V)$ assume that $U=V^T$ and $f(x)$ is a PSD kernel function. The quadratic lower bounds for relative error low-rank approximation of $f(U.V)$ for polynomial $f(.)$ show that both these assumptions are necessary for a subquadratic algorithm. This is an interesting result of broader interest to the sketching community. The lower bound proof techniques via reductions from OVP and MIP problems and bounding the leverage score of flat sparse vectors are also novel.\n\n2) The paper is generally easy to follow for the most part.\n\nWEAKNESSES:\n1) Apart from the lower bounds of Theorem 3.1 and 3.3, the other results seem to be straightforward extensions of these results or, for the case of the upper bounds, applications of existing algorithms and results in the literature. \n\n2) Though the results are interesting, the intuition behind some of the theoretical results presented in the paper are unclear. \n\n     a) Specifically, it might be useful to explain more in detail why is there a separation in the lower bound between the $U=V^T$ and $U \\neq V^T$ cases for general $p$.\n\n     b) It might be good to explain why the lower bound for general $p$ extend to more general functions with Taylor series of the form $f(x)=x^p+O()$ like in the case for odd $p$. Where is the bottleneck for general $p$?\n\n2) Though this is primarily a theoretical paper, in the context of Neurips, some experimental evaluation of the algorithms for LRA might be useful for understanding how well the algorithms perform well in practice. \n\nThere are also some typos and confusing notations in the paper:\n1) In the line 260, $Z$ has dimension $n \\times s$. The $s$ here is the rank of $U''$ and is different from the dimension of the  vectors $a_i$ and $b_i$.\n2) In the statement of Theorem 4.1, on the RHS, it seems the $U,V$ in argument $\\min_{U,V}$ should be changed to different notation as this is different from the input to the problem.\n\nQUESTIONS:\nOther than the points raised above, there doesn't seem to be any discussion on the lower bound for additive error so it is unclear how good the upper bound presented in Theorem 4.2 is. It might be useful to add a short discussion there.\n\nLIMITATIONS:\nI don't think there are any other limitations other than the ones mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
For two matrices U,V one is often interested in computing f(UV) for some entrywise function f. Several papers in the past decade considered the problem of approximating f(UV) by a product AB with small inner-dimension (low-rank approximation). While a number of past papers presented clever positive results, this paper is about demonstrating various negative results. I am very far from the field of complexity and cannot evaluate soundedness of the claims. However, I should say that the negative results are probably best presented in a different venue (SODA etc), since this conference is mostly interested in doing things rather than in enumerating all things that cannot be done.

STRENGTHS:
Key problem of contemp interest.

WEAKNESSES:
Complexity results are likely irrelevant to most attendees.

QUESTIONS:
n/a

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25