SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the latter case, we note that our reduction structure ensures that all entries in this vector have the same magnitude. Consequently, since the column span of the output has low rank, each such entry in the support of this sparse vector corresponds to a large leverage score, and so to find the support of this unknown sparse column we can use algorithms to compute or approximate all leverage scores given the low rank factorization of the entrywise transformed matrix. This enables us to find a small superset of the support of the sparse vector, and then brute-force which pairs of entries in the superset correspond to vectors in the original OVP problem that intersect. The overall time in the reduction is negligible compared to the time to solve OVP, and therefore it must be that finding the factorization of the entrywise transformed matrix itself was expensive.
In the former case, it follows that the column spans of the output are necessarily far from each other in the two cases, and we can quickly check this and therefore solve the OVP problem. Together, it follows that the time for finding the factorization of the entrywise transformed matrix itself must have been large. While there are quantitative differences in the two cases of odd and even integers  $p$ , the proofs both follow this strategy.
Algorithm 1 OVP to LRA Reduction
Input: Sets  $\mathbf{A} = \{a_1, \dots, a_n \in \{0, 1\}^s\}$  and  $\mathbf{B} = \{b_1, \dots, b_n \in \{0, 1\}^s\}$
Output: YES if there exists  $a^\top b = 0$  and NO otherwise.
1: Choose  $c \in \{-1, 1\}^n$  uniformly at random
2: Let  $\mathbf{U} = [\mathbf{A}|c] \in \mathbb{R}^{n \times (s + 1)}$  and  $\mathbf{V}^{\top} = [\mathbf{B}|c] \in \mathbb{R}^{n \times (s + 1)}$  and  $r = s + 1$ .
3: Let  $\mathbf{W} = \mathrm{LRA}(\mathbf{U},\mathbf{V},f = |x|^p)$  for constant relative error and  $\alpha$  additive error
4: Compute  $\mathbf{U}'' = \mathbf{U}\otimes \ldots \otimes \mathbf{U}\in \mathbb{R}^{n\times r^p}$ ,  $\mathbf{U}$  tensored with itself  $p$  times and similarly for  $\mathbf{V}''$ .
5: Let  $r_i^2 = \|\mathbf{U}''\mathbf{V}''e_i\|_2^2 - \|\mathbf{U}''\mathbf{V}''\mathbf{W}\mathbf{W}^\top e_i\|_2^2$  for  $i$ -th column  $\triangleright$  Calculate column distances of  $\mathbf{U}''\mathbf{V}''$  to column span of  $\mathbf{W}$ .
6: If any of  $r_i^2 > 1.01\alpha$ , output YES  $\triangleright \alpha$  is any upper bound on additive error of LRA
7: Compute  $1 / 2$ -approximate leverage scores  $\ell_{i}$  of  $\mathbf{U}''$  and let  $S = \{i\in [n]\mid \ell_i\geq 1 / (100n^{o(1)})\}$ .  
 $\triangleright$  Finds the representative subset and the threshold is given by the OVP assumption.
8: Compute all dot products between  $a_{i}$  and  $\mathbf{B}$  for all  $i\in S$ . Output YES if there exists a pair such that  $a_i^\top b_j = 0$  and NO otherwise
3.1 General Functions of  $|x|^{p}$  for odd integers  $p$
In the following theorem we will also allow for a tiny amount of additive error, as this will be useful for our later lower bound applications where we approximate other functions using a Taylor series and reduce from the following theorem.
Theorem 3.1. Suppose  $n \in \mathbb{N}$ ,  $f(x) = |x|^p$  for an odd integer constant  $p = O(\log(n))$ , and  $r = O(\log(n))$ . There is a positive integer  $k = n^{o(1)}$ , such that for any approximation factor  $\Delta \geq 1$  and any constant  $\alpha < 2$ , any possibly randomized, algorithm which given any input  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$  outputs  $W \in \mathbb{R}^{n \times k}$  satisfying
$$
\left\| f (U \cdot V) W W ^ {\top} - f (U \cdot V) \right\| _ {F} ^ {2} \leq \Delta \cdot \left\| [ f (U \cdot V) ] _ {k} - f (U \cdot V) \right\| _ {F} ^ {2} + \alpha ,
$$
with constant probability, requires  $(nd)^{1 - o(1)}$  time, under Assumption 1. Further, this holds even if  $U = V^T$ . Here  $[f(U \cdot V)]_k$  denotes the best rank- $k$  approximation to  $f(U \cdot V)$  in the Frobenius norm.
Proof. Suppose  $A$  and  $B$  are input sets to the OVP problem of Assumption 1 with parameter  $r = s + 1$ , where  $s$  is the dimension of the points. We let the rows of  $U$  be the points in  $A$ , but we append one additional dimension, represented as column vector  $c$  to  $U$  that is chosen uniformly at random in  $\{-1, 1\}^n$ . Similarly, the columns of  $V$  correspond to the points in  $B$ , but we append one additional row to  $V$ , which is equal to  $c^T$ . Observe that if  $A = B$ , then necessarily  $U = V^\top$ .
Let Case 1 be when there is no  $a \in A$  and  $b \in B$  with  $a^T b = 0$ , and Case 2 be when there is an  $a \in A$  and a  $b \in B$  with  $a^T b = 0$ , and we are deciding whether we are in Case 2. We claim that our algorithm (Algorithm 1) will always output NO when we are in Case 1 and will output YES with constant probability when we are in Case 2. This clearly suffices to solve OVP with constant probability.
Notice that if there is no  $a \in A$  and  $b \in B$  for which  $a^T b = 0$ , then for all  $a \in A$  and  $b \in B$ , we have  $a^T b \geq 1$ . Consequently with probability 1 over the choice of column vector  $c$ , for all  $i \in [n]$  and  $j \in [d]$ ,  $U_i V_j \geq 0$ , where  $U_i$  is the  $i$ -th row of  $U$  and  $V_j$  is the  $j$ -th column of  $V$ . Consequently,  $f(U_i V_j) = (U_i V_j)^p$ . Thus,  $f(U \cdot V) = U'' V''$ , where each row of  $U'' \in \mathbb{R}^{n \times r^p}$  is the Khatri-Rao product of itself  $p$  times, and each column of  $V'' \in \mathbb{R}^{r^p \times d}$  is the Khatri-Rao product of itself  $p$  times.
Observe that the rank of  $f(U \cdot V)$  is at most  $\mathrm{rank}(U'') + n^{o(1)} \leq (s + 1)^p + n^{o(1)}$  since by Remark 3 we have at most  $n^{o(1)}$  dot product pairs that are zero, implying that  $f(U \cdot V)$  can be obtained from  $U'' \cdot V''$  by changing at most  $n^{o(1)}$  entries. It follows that there is a value  $k \leq n^{o(1)} + (s + 1)^p$  such that, in both cases, for any multiplicative approximation factor  $\Delta \geq 1$ , necessarily the output  $W$  satisfies  $\| f(U \cdot V)WW^\top - f(U \cdot V)\|_F^2 \leq \alpha$  with constant probability.
Next, we compare the column span of  $W$  to that of  $U^{\prime \prime} \cdot V^{\prime \prime}$ . In Case 1, there is no pairwise dot product that is zero, so we have that  $f(U \cdot V) = U^{\prime \prime}V^{\prime \prime}$ . Therefore, each column of  $U^{\prime \prime} \cdot V^{\prime \prime}$  has squared distance at most  $\alpha$  to the column span of  $W$ .
Note that we can compute all squared distances of  $U''V''$  to the column span of  $W$  in  $O(n^{1 + o(1)})$  time. This is so that we can detect whether we are in case 1, where  $U''V''$  has at squared distance at most  $\alpha$  to the column span of  $W$ , or we are in case 2. Specifically, we compute the squared distance by the Pythagorean theorem and since  $W$  is orthogonal:
$$
r _ {i} ^ {2} = \left\| U ^ {\prime \prime} \cdot V ^ {\prime \prime} e _ {i} \right\| _ {2} ^ {2} - \left\| U ^ {\prime \prime} V ^ {\prime \prime} W W ^ {\top} e _ {i} \right\| _ {2} ^ {2}
$$
and for each  $i$ , we can compute this in  $n^{o(1)}$  time, given the matrices we have precomputed in the previous paragraph. Thus, we can compute all squared distances up to additive  $1 / \mathrm{poly}(n)$  in  $n^{1 + o(1)}$  time. If we see that any squared distance is larger than  $\alpha + 1 / \mathrm{poly}(n)$ , we know we are in Case 2.
Therefore, we may assume in what follows that each column of  $U'' \cdot V''$  has squared distance at most  $\alpha + 1 / \mathrm{poly}(n)$  from the column span of  $W$ . In this case we can also determine in  $n^{1 + o(1)}$  time, with constant probability, whether we are in Case 2. To do so, we apply a leverage score approximation algorithm [Clarkson and Woodruff, 2013] to find a small representative subset  $A' \subset A$  such that  $|A'| = n^{o(1)}$  and if there exists  $a \in A, b \in B$  such that  $a^\top b = 0$ , then  $a \in A'$ . Specifically, suppose there is an  $a \in A$  and a  $b \in B$  for which  $a^T b = 0$ , and suppose  $U_i$  extends  $a$  by one coordinate and  $V_j$  extends  $b$  by one coordinate. Then with probability at least  $1/2$ ,  $U_i^T V_j = -1$ , and so  $f(U_i^T V_j) = 1$ . Let us condition on this event. Then we have the following claim in this case.
Claim 6. Let  $a \in A$  and  $b \in B$  be such that  $a^T b = 0$ , and the corresponding row  $U_i$  and column  $V_j$  satisfy  $U_i^\top V_j = -1$ . Also, suppose each column of  $U' \cdot V'$  has squared distance at most  $\alpha + 1 / \text{poly}(n)$  from the column span of  $U''$ . Then, in  $n^{1 + o(1)}$  time, we can find a representative subset  $A' \subset A$  of size  $n^{o(1)}$  such that  $a \in A'$  with high probability.
Proof. (of Claim) First note that by Remark 3, there can be at most  $n^{o(1)}$  coordinates in the  $j$ -th column of  $f(U \cdot V)$  which differ from their corresponding coordinate value in the  $j$ -th column of  $U''V''$ , since each such difference corresponds to a pair of points  $a \in A$  and  $b \in B$  in the OVP problem for which  $a^T b = 0$ . Hence, since  $f(U \cdot V)$  has squared distance at most  $\alpha + 1 / \mathrm{poly}(n)$  from the column span of  $W$ , then consider the following vectors by taking the difference of the  $j$ -th column of  $f(U \cdot V)$  and the  $j$ -th column of  $U'' \cdot V''$ , we would have vectors  $v$  denoting the difference vector and  $e$  being the residual vector of  $v$  projected on the column span of  $U''$ , with the following properties:
-  $v$  contains at most  $n^{o(1)}$  values, each of value equal to 2,
$\| e\| \leq \alpha +1 / \mathrm{poly}(n)$
-  $v + e$  is in the column span of  $U''$ .
Let  $S \subset [n]$  be the set of indices  $i$  for which  $\frac{(v + e)_i^2}{\|v + e\|_2^2} \geq 1 / n^{o(1)}$ . Since  $\alpha + 1 / \mathrm{poly}(n)$  is at most a constant strictly less than 2, it follows that for each  $i$  in the support of  $v$ , since  $v_i = 2$ , we have that  $\frac{(v + e)_i^2}{\|v + e\|_2^2} \geq 1 / n^{o(1)}$ . Since also  $\| v + e \|_2^2 \leq n^{o(1)}$ , the support of  $v$  is included in  $S$  and  $|S| \leq n^{o(1)}$ .
Recall the definition of leverage scores  $\ell_{i}$  from Section 2 and that the sum of the  $n$  leverage scores of  $B\in \mathbb{R}^{n\times t}$  is exactly equal to the rank of  $B$ , which is at most  $t$ . Since  $v + e$  is in the column span of  $U''$ , the  $i$ -th leverage score of  $U''$  satisfies  $\ell_{i}(U'')\geq (v + e)_{i}^{2} / \| v + e\|_{2}^{2} = \Omega (1 / n^{o(1)})$ .
Consequently, each coordinate  $i$  in  $S$  satisfies  $\ell_i(U'') = \Omega(1/n^{o(1)})$ . It is known [Clarkson and Woodruff, 2013] how, given a matrix  $B$ , in  $O(nt\log n) + t^{O(1)}$  time, one can compute a list  $\ell_1', \ldots, \ell_n'$  with  $\ell_i' = \Theta(\ell_i)$  for all  $i$  with probability  $1 - 1/n^{100}$ . Consequently, using that  $\mathrm{rank}(U'') \leq r^p$ , given  $U''$  one can find a superset  $T$  containing  $S$  for which  $|T| = O(r^p n^{o(1)})$  in  $n^{1 + o(1)}$  time, where recall  $r = s + 1$  and  $s = n^{o(1)}$  and  $p$  is constant. Note that our bound on  $|T|$  follows since we just need to keep the leverage scores that are  $\Omega(1/n^{o(1)})$  and the sum of all leverage scores is at most  $r^p = n^{o(1)}$ , recalling  $r = s + 1$  and  $s = n^{o(1)}$ .
Continuing the proof of Theorem 3.1, by our claim, when we are in Case 2, with at least constant probability, we find in time  $O(n^{1 + o(1)})$  a representative subset  $A'$  of size  $O(n^{o(1)})$ . Given our subset  $A'$ , one can then compute all pairs of dot products between the points in  $A'$  and the points in  $B$  in  $O(n^{o(1)}ds)$  time, and since we may assume  $n \geq d$  without loss of generality, we can compute all such pairs in  $n^{1 + o(1)}$  time. Lastly, if no such  $a, b$  exist, when we are in Case 1, this algorithm cannot err. Therefore, we have an algorithm that can solve  $OVP$ , which then violates Assumption 1, using that  $(nd)^{1 - o(1)} = n^{1 + \Omega(1)}$ .
Our lower bound techniques also apply to a number of important functions  $f$  that do not have the form of  $|x|^p$  for an integer  $p$ . We show that our lower bounds for LRA in fact hold for any function  $g(x) = f(|x|)$ , where  $f(x)$  is a function that admits a Taylor expansion with a dominant term of  $x^p$ , for odd  $p$ , around 0. Of particular interest in natural language processing [Liang et al., 2020, Jiang et al., 2021] is the function  $g(x) = \log (1 + |x|)$ , where satisfies  $g(x) = f(|x|)$ , where  $f(x) = x + O(|x|^2)$  and thus we can appeal to our lower bound with  $p = 1$ .
Theorem 3.2. Suppose  $n \in \mathbb{N}$ ,  $r = O(\log(n))$ , and  $g(x) = f(|x|)$ , where  $f$  admits a Taylor expansion  $f(x) = c_p x^p + O(|x|^{p+1})$  for odd  $p = O(\log(n))$ . There is a positive integer  $k = n^{o(1)}$  such that for any approximation factor  $\Delta \geq 1$ , any possibly randomized, algorithm which given inputs  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$  outputs  $W \in \mathbb{R}^{n \times k}$  satisfying
$$
\left\| g (U \cdot V) W W ^ {\top} - g (U \cdot V) \right\| _ {F} ^ {2} \leq \Delta \cdot \left\| [ g (U \cdot V) ] _ {k} - g (U \cdot V) \right\| _ {F} ^ {2}
$$
with constant probability, requires  $(nd)^{1 - o(1)}$  time, under Assumption 1. Further, this holds even if  $U = V^{\top}$ .
3.2 Polynomials of All Integers  $p$
We now give the proof for  $p$ -degree polynomials for all integers  $p$ , which is weaker than Theorem 3.1 for odd integers, but this is the first such lower bound for even integers. The weaker bound cannot be substantially improved since in this setting  $f(x) = x^p$ , so  $f(UV)$  is in fact a kernel and matrix vector multiplication can be performed in  $O(nr^p)$  time. For intuition why our lower bound techniques do not extend, recall that our previous reduction forces the absolute value operation to essentially alter entries of  $(UV)^p$  but only at entries of the original with zero dot product. Therefore, we can write as a sum of a low rank matrix (from tensor product) and a sparse matrix, whose sparse entries now represent the OVP pairs. However, when  $p$  is even, the absolute value operation leaves the entries unchanged and does not induce the additional sparse matrix. Therefore, our lower bounds for this setting rely on a slightly different variant of OVP, specifically quadratic lower bounds for finding the maximum dot product (Assumption 4).
Intuitively, our new reduction is as follows: Let  $\mathsf{OPT}$  be the maximum inner product and by assumption, consider this small set of large inner product pairs, which represents a small number of entries in that are large in magnitude so that when you apply a threshold at  $\mathsf{OPT} / 100$ , the resulting matrix is sparse. Since  $f(x) = x^{p}$  amplifies the magnitude differences, it follows that  $(UV)^{p}$  is much closer relatively to an approximately sparse, and therefore low rank, matrix. Therefore an approximate low rank approximation (LRA) algorithm can recover this sparse low rank matrix well enough so that the span of the approximate matrix can be used, via leverage score computations, to solve the APX-Max-IP problem.
Theorem 3.3. Let  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$  be given with  $r = O(\log(n))$ . Suppose  $f(x) = x^p$  for an integer  $p \geq 1$ . There is a positive integer  $k = n^{o(1)}$ , such that for any approximation factor  $\Delta \geq 1$ , any, possibly randomized, algorithm which outputs  $W \in \mathbb{R}^{n \times k}$  satisfying  $\| f(U \cdot V)WW^\top - f(U \cdot V)\|_F^2 \leq \Delta \cdot \| [f(U \cdot V)]_k - f(U \cdot V)\|_F^2$ , with constant probability for any constant  $\Delta > 1$ , requires  $\min((nd)^{1 - o(1)}, 2^{\Omega(p)})$  time, under Assumption 4.
The lower bound for general functions  $f(x)$  in Theorem 3.2 with odd-degree dominating term in its Taylor expansion works because we can scale down the entries in our input matrix so that our relative-error approximation guarantee is still preserved but the function is largely approximated by only the leading polynomial term. By a combination of the same taylor expansion argument with a slight more general version of Theorem 3.3 that handles small additive error, we note that we can extend our results to when  $p$  is even.
We note that our lower bounds do not generalize to the case when  $U = V$  because as mentioned in our intuitive introduction, our reduction hinges on the sparse low-rank structure of  $(UV)^p$ . However, when  $U = V$ , the sparsity structure is broken as the diagonal of the matrix is now larger than the maximum inner product of two different vectors, and this destroys the low rank structure. In some sense, the diagonal of our matrix is forced to include the dot product of  $u_i$  with itself and this shifts the entire matrix by a large multiple of the identity, crucially removing the sparse + low-rank structure that we exploited in our lower bound argument before. Indeed, as we remark below, the resulting positive semidefinite (PSD) structure allows us to derive a fast LRA algorithm and our lower bound no longer holds in this setting.
Remark 7. Recall that when  $U = V^{\top}$ , there is an  $nr(k / \epsilon)^{\omega -1}$  subquadratic time algorithm when  $k = O(n^{o(1)})$ , for  $(1 + \epsilon)$ -relative error approximation, but reducing the  $U \neq V^{\top}$  case to the  $U = V^{\top}$  case requires blowing up  $k$  to  $k + r^{p}$ . Therefore, our lower bounds imply that the  $U \neq V^{\top}$  case is strictly more difficult in terms of runtime in certain settings.
4 LRA Algorithms from Matrix Vector Products
In this section, we can show upper bounds for low rank approximation of entrywise transformed products that are  $O(n^{1 + o(1)})$  when  $f(x)$  represents a kernel matrix but  $U \neq V^{\top}$ . Specifically, we focus on polynomial functions of the form  $f(x) = x^p$  and demonstrate that our lower bounds are tight for relative error LRA. Note that we have shown that low rank approximation guarantees for PSD matrices when  $U = V^{\top}$  cannot translate to the case when  $U \neq V^{\top}$ . Still, we demonstrate that relative error low rank approximation is possible in  $n^{1 + o(1)}$  time for polynomial kernels with even degree, although there will be an exponential dependence on  $p$  for relative error low rank approximation. Our relative error algorithms are relatively standard and rely on low rank projections by using matrix vector products to perform randomized sketching to reduce our row or column dimension to  $O(\mathrm{poly}(k / \epsilon))$  [Woodruff, 2014].
Theorem 4.1. Let  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$ . Suppose  $f(x) = x^p$  for an even integer  $p \geq 1$  and  $k < r^p$ . For any approximation factor  $\epsilon > 0$ , there is an algorithm that outputs  $U' \in \mathbb{R}^{n \times k}$  and  $V' \in \mathbb{R}^{k \times d}$  satisfying  $\| U' \cdot V' - f(U \cdot V)\|_F^2 \leq (1 + \epsilon) \cdot \| [f(U \cdot V)]_k - f(U \cdot V)\|_F^2$  with constant probability with runtime  $O((n + d)r^p k / \epsilon^3 + poly(r^p / \epsilon))$ .
Proof. Note that  $f(U_i V_j) = (U_i V_j)^p$ . Thus, we may rewrite  $f(U \cdot V) = U'' V''$ , where each row of  $U'' \in \mathbb{R}^{n \times r^p}$  is the Khatri-Rao product of itself  $p$  times, and each column of  $V'' \in \mathbb{R}^{r^p \times d}$  is the Khatri-Rao product of itself  $p$  times. Note that the rank of  $U'' V''$  is at most  $d^p$ .
Let  $S$  be a random Gaussian sketching matrix with  $O(k / \epsilon)$  rows, so we know that these matrices satisfy the  $(\sqrt{\epsilon / k}, 9 / 10, l)$ -JL property [Woodruff, 2014]. Furthermore, let  $R$  be a random Gaussian matrix with  $O(\min(k / \epsilon^3, r^p / \epsilon^2))$  columns, so we know that it is a  $(1 + O(\epsilon)) \ell_2$  subspace embedding of the row space of  $SU''$ . Then, by Theorem 47 of Woodruff [2014], the following is true with constant probability
$$
\| (U ^ {\prime \prime} V ^ {\prime \prime} R) (S U ^ {\prime \prime} V ^ {\prime \prime} R) ^ {+} (S U ^ {\prime \prime} V ^ {\prime \prime}) - f (U V) \| _ {F} ^ {2} \leq (1 + \epsilon) \cdot \| [ f (U \cdot V) ] _ {k} - f (U \cdot V) \| _ {F} ^ {2}
$$
Finally we bound the runtime of computing this product. Note that we may compute  $SU''$  and  $V''R$  in  $nr^p \cdot (k / \epsilon + \min(k / \epsilon^3, r^p / \epsilon^2))$  time. Then, note that the remaining products can be computed in
$\mathrm{poly}(kr^p/\epsilon) = \mathrm{poly}(r^p/\epsilon)$  time. And lastly, the rank  $k$  approximation follows from solving for the low rank approximation on the restricted subspace  $SU''V''R$  of rank and letting  $Z = [U''V''RU]_k$ , where  $U$  is the orthonormal basis such that  $UU^\top$  is the projection onto the row space of SAR.
We also provide an additive error guarantee on the low rank approximation guarantees that depends polynomially on  $p$ , as opposed to the tight exponential dependence in the relative error setting. This implies that additive error LRA is strictly easier and can be accomplished in subquadratic time when  $p = \Omega(\log(n))$ . Note that lower bounds for additive error are not emphasized in this paper as the upper bound is already quite competitive and simply outputting requires  $\Omega(n * \mathrm{poly}(p))$ .
Our guarantees follow from tensor-based sketching techniques that are applied along each tensor dimension to remove the exponential dependence on  $p$ . We note that this results in an additive error term that is on the order of the  $p$ -norms of  $U, V$ . This term is related to the absolute error guarantees of [Han et al., 2020], which also depends on the product of the  $p$ -norms of Euclidean norms of the rows of  $U, V$ .
Algorithm 2 TensorSketch LRA
Input: Matrices  $\mathbf{U}\in \mathbb{R}^{n\times r},\mathbf{V}\in \mathbb{R}^{r\times d},p > 0$  even integer   
Output: rank  $k$  approximation of  $f(UV)$  , where  $f(x) = |x|^p$    
1: Let T be a TensorSketch with  $m = O(p\epsilon^{-2})$  rows See Ahle et al. [2020]   
2: Compute  $\mathbf{U}^{\prime \prime \prime} = \mathbf{U}^{\prime \prime}\mathbf{T}^{\top}$  and  $\mathbf{V}^{\prime \prime \prime} = \mathbf{T}\mathbf{V}^{\prime \prime}$  , where  $\mathbf{U}^{\prime \prime}\in \mathbb{R}^{n\times r^p}$ $\mathbf{V}''\in \mathbb{R}^{r^p\times d}$  are U, V tensored themselves  $p$  times.   
3: Let S, R be random Gaussian matrices with  $O(k / \epsilon)$  rows and  $O(p / \epsilon^4)$  columns respectively   
4: Compute an orthonormal basis P of row span of SU'V''R. This implies PP= (SU'V''R)+(SU'V''R)   
5: Compute the rank  $k$  decomposition:  $\mathbf{U}'\mathbf{V}' = [\mathbf{U}''\mathbf{V}''\mathbf{RP}]_k.$ $\triangleright [\mathbf{M}]_k$  is the best rank k approximation to M   
6: Output U', V'P'(SU''V''R)+SU''V''
Theorem 4.2. Let  $U \in \mathbb{R}^{n \times r}$  and  $V \in \mathbb{R}^{r \times d}$ . Suppose  $f(x) = x^p$  for an even integer  $p \geq 1$  and  $k < r^p$ . For any approximation factor  $\epsilon > 0$ , there is an algorithm (Algorithm 2) that outputs  $U' \in \mathbb{R}^{n \times k}$  and  $V' \in \mathbb{R}^{k \times d}$  satisfying  $\| U' \cdot V' - f(U \cdot V)\|_F^2 \leq (1 + \epsilon) \cdot \| [f(U \cdot V)]_k - f(U \cdot V)\|_F^2 + \epsilon^2 L^2$  with constant probability with runtime  $O((n + d) \cdot \text{poly}(p, r, k, 1 / \epsilon))$ , where the additive term is given by  $L^2 = (\sum_{i=1}^{n} \| U_i\|_2^{2p})(\sum_{i=1}^{d} \| V_i\|_2^{2p}) = \| U\|_{2p,2}^{2p} \| V^\top \|_{2p,2}^{2p}$ .
4.1 Lower Bounds on Matrix Multiplication